開発環境では、**DB を壊さない前提で、migration → API → 画面 → 同期 → ZIP → 履歴**の順に見るのがよいです。今回の差分は素材管理全体に触っているので、単体でチョンチョン見るより、素材の一生を通すのが早いです。
## 0. 先に方針
**やらないこと:**
```sh id="snb3i6"
rails db:drop
rails db:reset
rails db:setup
DISABLE_DATABASE_ENVIRONMENT_CHECK=1 ...
```
これは禁止。
開発 DB に本番データを入れているなら、床板を剥がして耐震確認するようなものです。
---
## 1. migration 確認
まず現在の状態を見る。
```sh id="qsdfpt"
cd backend
RAILS_ENV=development bundle exec rails db:migrate:status
```
その後、通常 migration。
```sh id="a89fir"
RAILS_ENV=development bundle exec rails db:migrate
```
見るポイント:
```txt id="c1u57a"
materials に source_* / normalized_source_key / version_no がある
material_versions に event_type / file snapshot / source snapshot がある
material_export_items がある
material_sync_suppressions がある
material_sync_sources がある
既存 materials に material_versions version_no=1 create が backfill されている
```
確認用:
```sh id="g4vd4m"
RAILS_ENV=development bundle exec rails runner '
puts "materials=#{Material.count}"
puts "versions=#{MaterialVersion.count}"
puts "materials without versions=#{Material.left_joins(:material_versions).where(material_versions: { id: nil }).count}"
puts "sync suppressions table=#{ActiveRecord::Base.connection.table_exists?(:material_sync_suppressions)}"
'
```
ここで `materials without versions=0` になれば、backfill は通っています。
---
## 2. 既存素材一覧の画面確認
フロントを起動して `/materials` を見る。
```sh id="vl28jd"
cd frontend
npm run dev
```
見る観点:
```txt id="i2ovxw"
初期表示で素材が出る
初期表示ではグルーピングがオフ
タグなし素材も出る
カード表示でサムネまたは代替テキストが出る
一覧表示に切り替えられる
q / tag_state / media_kind / sort / direction が効く
```
ここでまず、普通の素材一覧が壊れていないことを確認します。
---
## 3. 左タグバーの確認
`/materials` を開いて左タグバーからタグを選ぶ。
見る観点:
```txt id="n6udul"
URL が tag_id=...&include_descendants=1&group_by=parent_tag になる
選択中タグが左バーで強調される
一覧上部に「選択中」の表示が出る
「タグ選択を解除」で通常表示に戻れる
解除後、tag_id / include_descendants / group_by / page が消える
子タグ・孫タグの素材も一覧に出る
親タググルーピングされる
```
特に重要なのはこれ。
```txt id="d33os0"
親タグ A を選択
A に直接紐づく素材
A > B に紐づく素材
A > B > C に紐づく素材
が同じ一覧に出ること
```
---
## 4. 選択タグから素材追加
左タグからタグを選択した状態で、一覧上部の **このタグに素材を追加** を押す。
見る観点:
```txt id="qdd5uw"
素材追加画面の tag 欄に選択中タグ名が初期入力されている
file または url を指定して保存できる
保存後 return_to で元のタグ選択済み一覧に戻る
戻った一覧に追加した素材が出る
material_versions に create が 1 件できる
```
Rails console でも確認できます。
```sh id="i07rrb"
RAILS_ENV=development bundle exec rails runner '
m = Material.order(id: :desc).first
puts({ id: m.id, tag: m.tag&.name, versions: m.material_versions.count, version_no: m.version_no }.inspect)
'
```
---
## 5. グループ見出しから素材追加
親タググルーピング表示中に、各グループ見出しの **このタグに素材を追加** を押す。
見る観点:
```txt id="fpx8w4"
グループタグ名が tag 欄に初期入力される
保存後、元の一覧に戻る
追加素材がそのグループ内に出る
```
ここは今回の導線の肝です。棚の見出しから直接その棚へ素材を置けるかを見る。
---
## 6. 素材更新と履歴
既存素材の詳細または編集導線から、タグ・URL・ファイル・export path を更新する。
見る観点:
```txt id="w36i61"
更新前 snapshot が無ければ create が補われる
更新後 update version ができる
file_blob_id / file_filename / file_sha256 が material_versions に入る
export_paths_json が履歴に残る
/materials/changes または /materials/versions で履歴が見える
```
console:
```sh id="a3okhv"
RAILS_ENV=development bundle exec rails runner '
m = Material.order(updated_at: :desc).first
puts m.material_versions.order(:version_no).map { |v|
[v.version_no, v.event_type, v.tag_name, v.file_filename, v.file_sha256, v.export_paths_hash]
}.inspect
'
```
---
## 7. サムネイル
画像素材を追加して、一覧にサムネイルが出るか確認。
動画素材があるなら、`ffmpeg` が入っている環境で backfill。
```sh id="qxm8fj"
cd backend
RAILS_ENV=development bundle exec rails materials:thumbnails:backfill
```
見る観点:
```txt id="s9a1vf"
画像は 180x180 のサムネが付く
動画はフレームからサムネが作られる
非対応ファイルは代替テキスト表示になる
ログに result が出る
```
---
## 8. ZIP export
export path がある素材を用意して、ブラウザで確認。
```txt id="aq7f7o"
/materials/download.zip?profile=legacy_drive
```
見る観点:
```txt id="h4rve5"
ZIP が落ちる
entry path が material_export_items.export_path になる
disabled な export item は入らない
ファイル実体が欠けている場合は 422 と missing_files が返る
```
---
## 9. 抑止
`/materials/suppressions` で path prefix 抑止を追加する。
見る観点:
```txt id="w80jbu"
member で作成できる
guest は forbidden / unauthorized
google_drive_path_prefix で既存素材が discard される
discard 履歴が material_versions に残る
同期時に同じ source_path 配下が再作成されない
```
---
## 10. Google Drive 同期
開発環境では、まず小さいフォルダでやるのがよいです。
いきなり本番素材集フォルダを食わせると、ログが藪になります 🌿
必要な ENV:
```sh id="kzbb7f"
GOOGLE_DRIVE_SERVICE_ACCOUNT_EMAIL=...
GOOGLE_DRIVE_PRIVATE_KEY_PATH=...
MATERIAL_SYNC_SOURCE_KIND=google_drive_path
MATERIAL_SYNC_SOURCE_FILE_ID=<folder_id>
MATERIAL_SYNC_SOURCE_NAME=dev-small-folder
MATERIAL_SYNC_SOURCE_PROFILE=legacy_drive
```
seed で source 作成、または console で作成。
```sh id="sec40c"
RAILS_ENV=development bundle exec rails db:seed
RAILS_ENV=development bundle exec rails materials:sync
```
見る観点:
```txt id="a5ltqu"
imported / updated / unchanged / suppressed / failed がログに出る
2 回目実行で unchanged が増える
tag は nil のままでも保存できる
人手で tag / url を付けた既存同期素材が、再同期で消えない
Google native file は skip される
download 後 sha256 block が効く
```
---
## 11. schema.rb は別途確認
これはテストというより merge gate です。
今回まだ怪しいので、差分に以下が混ざっていないことを確認します。
```txt id="s3k6da"
wiki_assets 削除
wiki_pages.next_asset_no 削除
素材管理と無関係な CHECK constraint 削除
素材管理と無関係な index order 消失
```
ここが残るなら、機能テストが通っても merge は止めた方がいいです。
---
## 最小テスト順
時間がないなら、この順で十分です。
```txt id="c1k2pw"
1. db:migrate
2. materials without versions = 0 を確認
3. /materials 初期表示
4. 左タグ選択 → 子孫込み表示 → グルーピング
5. タグ選択解除
6. 選択タグから素材追加 → return_to で戻る
7. グループ見出しから素材追加
8. 更新して material_versions を確認
9. ZIP export
10. 小さい Drive folder で materials:sync を 2 回
```
これで、今回の差分の主要な導線はほぼ踏めます。
Reviewed-on: #381
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
このコミットはPull リクエスト #381 でマージされました．
このコミットが含まれているのは:
@@ -0,0 +1,233 @@
|
||||
require 'json'
|
||||
require 'jwt'
|
||||
require 'net/http'
|
||||
require 'openssl'
|
||||
require 'tempfile'
|
||||
require 'uri'
|
||||
|
||||
|
||||
module GoogleDrive
|
||||
class ApiClient
|
||||
DRIVE_ENDPOINT = 'https://www.googleapis.com/drive/v3'
|
||||
TOKEN_ENDPOINT = 'https://oauth2.googleapis.com/token'
|
||||
FOLDER_MIME_TYPE = 'application/vnd.google-apps.folder'
|
||||
NATIVE_FILE_MIME_TYPE_PREFIX = 'application/vnd.google-apps.'
|
||||
DRIVE_SCOPE = 'https://www.googleapis.com/auth/drive.readonly'
|
||||
|
||||
def initialize service_account_email: ENV['GOOGLE_DRIVE_SERVICE_ACCOUNT_EMAIL'],
|
||||
private_key: ENV['GOOGLE_DRIVE_PRIVATE_KEY'],
|
||||
private_key_path: ENV['GOOGLE_DRIVE_PRIVATE_KEY_PATH'],
|
||||
subject: ENV['GOOGLE_DRIVE_SUBJECT']
|
||||
@service_account_email = service_account_email.to_s
|
||||
@private_key = load_private_key(private_key, private_key_path)
|
||||
@subject = subject.to_s.presence
|
||||
@access_token = nil
|
||||
end
|
||||
|
||||
def list_material_files_under_folder folder_id
|
||||
files = []
|
||||
each_material_file_under_folder(folder_id) { |entry| files << entry }
|
||||
files
|
||||
end
|
||||
|
||||
def each_material_file_under_folder folder_id
|
||||
return enum_for(__method__, folder_id) unless block_given?
|
||||
|
||||
walk_folder(folder_id, nil) do |entry, relative_path|
|
||||
next if entry['mimeType'] == FOLDER_MIME_TYPE
|
||||
next if native_file?(entry['mimeType'])
|
||||
|
||||
yield build_file_entry(entry, relative_path)
|
||||
end
|
||||
end
|
||||
|
||||
def fetch_material_file file_id
|
||||
metadata = get_file(file_id)
|
||||
return nil if native_file?(metadata['mimeType'])
|
||||
|
||||
build_file_entry(metadata, metadata['name'])
|
||||
end
|
||||
|
||||
def download_to_tempfile file_id, filename:
|
||||
tempfile = Tempfile.new(['material-sync', File.extname(filename.to_s)])
|
||||
tempfile.binmode
|
||||
request_binary("/files/#{ file_id }",
|
||||
{ alt: 'media', supportsAllDrives: true }) do |chunk|
|
||||
tempfile.write(chunk.b)
|
||||
end
|
||||
tempfile.rewind
|
||||
tempfile
|
||||
rescue StandardError
|
||||
tempfile&.close!
|
||||
raise
|
||||
end
|
||||
|
||||
def extract_file_id value
|
||||
raw = value.to_s.strip
|
||||
return nil if raw.blank?
|
||||
return raw unless raw.include?('/')
|
||||
|
||||
uri = URI.parse(raw)
|
||||
return uri.query.to_s[%r{(?:^|&)id=([^&]+)}, 1] if uri.query.present?
|
||||
return uri.path[%r{/folders/([^/]+)}, 1] if uri.path.include?('/folders/')
|
||||
|
||||
uri.path[%r{/d/([^/]+)}, 1]
|
||||
rescue URI::InvalidURIError
|
||||
nil
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def walk_folder folder_id, prefix, &block
|
||||
list_children(folder_id).each do |entry|
|
||||
relative_path = MaterialSyncExportPath.build(prefix:,
|
||||
relative_path: entry['name'])
|
||||
if entry['mimeType'] == FOLDER_MIME_TYPE
|
||||
walk_folder(entry['id'], relative_path, &block)
|
||||
next
|
||||
end
|
||||
|
||||
block.call(entry, relative_path)
|
||||
end
|
||||
end
|
||||
|
||||
def build_file_entry entry, relative_path
|
||||
{ id: entry['id'],
|
||||
name: entry['name'],
|
||||
mime_type: entry['mimeType'],
|
||||
relative_path: MaterialSyncExportPath.normalize_path(relative_path),
|
||||
sha256_checksum: entry['sha256Checksum'],
|
||||
web_view_link: entry['webViewLink'],
|
||||
web_content_link: entry['webContentLink'] }
|
||||
end
|
||||
|
||||
def list_children folder_id
|
||||
files = []
|
||||
page_token = nil
|
||||
|
||||
loop do
|
||||
response =
|
||||
request_json('/files', {
|
||||
q: "'#{ folder_id }' in parents and trashed = false",
|
||||
fields: 'nextPageToken,files(id,name,mimeType,sha256Checksum,' \
|
||||
'webViewLink,webContentLink)',
|
||||
orderBy: 'folder,name',
|
||||
pageSize: 1000,
|
||||
supportsAllDrives: true,
|
||||
includeItemsFromAllDrives: true,
|
||||
pageToken: page_token }.compact)
|
||||
files.concat(response.fetch('files'))
|
||||
page_token = response['nextPageToken']
|
||||
break if page_token.blank?
|
||||
end
|
||||
|
||||
files
|
||||
end
|
||||
|
||||
def get_file file_id
|
||||
request_json("/files/#{ file_id }",
|
||||
fields: 'id,name,mimeType,sha256Checksum,webViewLink,webContentLink',
|
||||
supportsAllDrives: true)
|
||||
end
|
||||
|
||||
def request_json path, params = {}
|
||||
response = request(:get, path, params:)
|
||||
unless response.is_a?(Net::HTTPSuccess)
|
||||
raise "Google Drive API error: #{ response.code } #{ response.body }"
|
||||
end
|
||||
|
||||
JSON.parse(response.body)
|
||||
end
|
||||
|
||||
def request_binary path, params = {}
|
||||
request(:get, path, params:) do |response|
|
||||
unless response.is_a?(Net::HTTPSuccess)
|
||||
raise "Google Drive download error: #{ response.code } #{ response.body }"
|
||||
end
|
||||
|
||||
response.read_body do |chunk|
|
||||
yield chunk.b
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def request method, path, params: nil
|
||||
uri = URI(DRIVE_ENDPOINT + path)
|
||||
uri.query = URI.encode_www_form(params) if params.present?
|
||||
|
||||
klass =
|
||||
case method
|
||||
when :get
|
||||
Net::HTTP::Get
|
||||
else
|
||||
raise ArgumentError, "Unsupported Google Drive request method: #{ method }"
|
||||
end
|
||||
request = klass.new(uri)
|
||||
request['Authorization'] = "Bearer #{ access_token }"
|
||||
|
||||
Net::HTTP.start(uri.host,
|
||||
uri.port,
|
||||
use_ssl: true,
|
||||
open_timeout: 10,
|
||||
read_timeout: 60) do |http|
|
||||
if block_given?
|
||||
http.request(request) do |response|
|
||||
return yield response
|
||||
end
|
||||
else
|
||||
http.request(request)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def access_token
|
||||
@access_token ||= fetch_access_token
|
||||
end
|
||||
|
||||
def fetch_access_token
|
||||
raise 'GOOGLE_DRIVE_SERVICE_ACCOUNT_EMAIL is required' if @service_account_email.blank?
|
||||
raise 'Google Drive private key is required' if @private_key.blank?
|
||||
|
||||
payload = { iss: @service_account_email,
|
||||
scope: DRIVE_SCOPE,
|
||||
aud: TOKEN_ENDPOINT,
|
||||
exp: 1.hour.from_now.to_i,
|
||||
iat: Time.current.to_i }
|
||||
payload[:sub] = @subject if @subject.present?
|
||||
|
||||
assertion = JWT.encode(payload, @private_key, 'RS256')
|
||||
uri = URI(TOKEN_ENDPOINT)
|
||||
request = Net::HTTP::Post.new(uri)
|
||||
request.set_form_data(
|
||||
grant_type: 'urn:ietf:params:oauth:grant-type:jwt-bearer',
|
||||
assertion:)
|
||||
|
||||
response =
|
||||
Net::HTTP.start(uri.host,
|
||||
uri.port,
|
||||
use_ssl: true,
|
||||
open_timeout: 10,
|
||||
read_timeout: 30) do |http|
|
||||
http.request(request)
|
||||
end
|
||||
unless response.is_a?(Net::HTTPSuccess)
|
||||
raise "Google OAuth error: #{ response.code } #{ response.body }"
|
||||
end
|
||||
|
||||
JSON.parse(response.body).fetch('access_token')
|
||||
end
|
||||
|
||||
def load_private_key private_key, private_key_path
|
||||
raw = private_key.to_s
|
||||
raw = raw.gsub('\n', "\n") if raw.present?
|
||||
raw = File.read(private_key_path) if raw.blank? && private_key_path.present?
|
||||
return nil if raw.blank?
|
||||
|
||||
OpenSSL::PKey::RSA.new(raw)
|
||||
end
|
||||
|
||||
def native_file? mime_type
|
||||
mime_type.to_s.start_with?(NATIVE_FILE_MIME_TYPE_PREFIX)
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,57 @@
|
||||
require 'digest'
|
||||
require 'json'
|
||||
|
||||
class MaterialFileSha256
|
||||
def self.blob_metadata blob
|
||||
metadata = blob.metadata
|
||||
return metadata if metadata.is_a?(Hash)
|
||||
return { } if metadata.blank?
|
||||
|
||||
JSON.parse(metadata)
|
||||
rescue JSON::ParserError
|
||||
{ }
|
||||
end
|
||||
|
||||
def self.metadata_sha256 blob
|
||||
blob_metadata(blob)['sha256'].presence
|
||||
end
|
||||
|
||||
def self.assign_metadata_sha256! blob, sha256
|
||||
return if sha256.blank?
|
||||
|
||||
metadata = blob_metadata(blob)
|
||||
metadata['sha256'] = sha256
|
||||
blob.metadata = metadata
|
||||
blob.save! if blob.changed?
|
||||
end
|
||||
|
||||
def self.from_blob blob, allow_download: false
|
||||
sha256 = metadata_sha256(blob)
|
||||
return sha256 if sha256.present?
|
||||
return nil unless allow_download
|
||||
|
||||
begin
|
||||
blob.open do |file|
|
||||
sha256 = Digest::SHA256.file(file.path).hexdigest
|
||||
assign_metadata_sha256!(blob, sha256)
|
||||
sha256
|
||||
end
|
||||
rescue ActiveStorage::FileNotFoundError, ArgumentError => error
|
||||
Rails.logger.warn(
|
||||
"MaterialFileSha256.from_blob failed for blob_id=#{blob.id}: " \
|
||||
"#{error.class}: #{error.message}",
|
||||
)
|
||||
nil
|
||||
end
|
||||
end
|
||||
|
||||
def self.from_upload upload
|
||||
tempfile = upload&.tempfile
|
||||
return nil unless tempfile
|
||||
|
||||
tempfile.rewind
|
||||
Digest::SHA256.file(tempfile.path).hexdigest.tap do
|
||||
tempfile.rewind
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,7 @@
|
||||
class MaterialImportBlockMatcher
|
||||
def self.match_for_sha256 sha256
|
||||
return nil if sha256.blank?
|
||||
|
||||
MaterialImportBlock.find_by(match_kind: 'sha256', sha256:)
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,56 @@
|
||||
class MaterialSyncExportPath
|
||||
class << self
|
||||
def build prefix: nil, relative_path: nil, filename: nil, source_file_id: nil
|
||||
path = normalize_path(relative_path)
|
||||
path = fallback_filename(filename, source_file_id) if path.blank?
|
||||
|
||||
normalized_prefix = normalize_path(prefix)
|
||||
return path if normalized_prefix.blank?
|
||||
return normalized_prefix if path.blank?
|
||||
|
||||
[normalized_prefix, path].join('/')
|
||||
end
|
||||
|
||||
def normalize_path path
|
||||
raw = path.to_s.delete("\0").tr('\\', '/').strip
|
||||
raw = raw.sub(%r{\A[A-Za-z]:/+}, '')
|
||||
raw = raw.sub(%r{\A/+}, '')
|
||||
return nil if raw.blank?
|
||||
|
||||
segments = raw.split('/').filter_map do |segment|
|
||||
normalized = normalize_segment(segment)
|
||||
normalized.presence
|
||||
end
|
||||
return nil if segments.empty?
|
||||
|
||||
segments.join('/')
|
||||
end
|
||||
|
||||
def uniquify path, source_file_id
|
||||
normalized = normalize_path(path)
|
||||
return normalized if normalized.blank? || source_file_id.blank?
|
||||
|
||||
ext = File.extname(normalized)
|
||||
base = ext.present? ? normalized.delete_suffix(ext) : normalized
|
||||
"#{ base }--#{ source_file_id }#{ ext }"
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def fallback_filename filename, source_file_id
|
||||
name = normalize_path(filename)
|
||||
return name if name.present?
|
||||
return nil if source_file_id.blank?
|
||||
|
||||
source_file_id.to_s
|
||||
end
|
||||
|
||||
def normalize_segment segment
|
||||
cleaned = segment.to_s.delete("\0").tr('\\/', '_').strip
|
||||
return nil if cleaned.blank? || cleaned == '.'
|
||||
return '__' if cleaned == '..'
|
||||
|
||||
cleaned
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,259 @@
|
||||
require 'digest'
|
||||
|
||||
class MaterialSyncImporter
|
||||
Result = Struct.new(:material, :action, :suppressed, :suppression, keyword_init: true)
|
||||
|
||||
def self.import! attributes
|
||||
new(attributes).import!
|
||||
end
|
||||
|
||||
def initialize attributes
|
||||
@attributes = attributes
|
||||
end
|
||||
|
||||
def import!
|
||||
source = source_attributes
|
||||
key = normalized_source_key(source)
|
||||
raise ArgumentError, 'normalized_source_key is required for material sync' if key.blank?
|
||||
|
||||
suppression = suppression_for(source)
|
||||
if suppression
|
||||
return Result.new(material: nil,
|
||||
action: :suppressed,
|
||||
suppressed: true,
|
||||
suppression:)
|
||||
end
|
||||
|
||||
material =
|
||||
Material
|
||||
.unscoped
|
||||
.find_or_initialize_by(normalized_source_key: key)
|
||||
if unchanged?(material, source)
|
||||
return Result.new(material:,
|
||||
action: :unchanged,
|
||||
suppressed: false,
|
||||
suppression: nil)
|
||||
end
|
||||
|
||||
event_type =
|
||||
if material.new_record?
|
||||
:create
|
||||
elsif material.discarded?
|
||||
:restore
|
||||
else
|
||||
:update
|
||||
end
|
||||
uploaded_blob = uploaded_blob!
|
||||
if @import_block
|
||||
return Result.new(material: nil,
|
||||
action: :suppressed,
|
||||
suppressed: true,
|
||||
suppression: @import_block)
|
||||
end
|
||||
|
||||
Material.transaction do
|
||||
if material.persisted?
|
||||
MaterialVersionRecorder.ensure_snapshot!(material,
|
||||
created_by_user: @attributes[:updated_by_user])
|
||||
end
|
||||
material.assign_attributes(material_attributes_for(material).merge(source))
|
||||
material.discarded_at = nil if material.respond_to?(:discarded_at=)
|
||||
material.file.attach(uploaded_blob) if uploaded_blob
|
||||
material.save!
|
||||
upsert_export_item!(material)
|
||||
MaterialVersionRecorder.record!(material:,
|
||||
event_type:,
|
||||
created_by_user: @attributes[:updated_by_user])
|
||||
end
|
||||
MaterialThumbnailGenerator.generate!(material)
|
||||
|
||||
Result.new(material:,
|
||||
action: event_type == :create ? :imported : :updated,
|
||||
suppressed: false,
|
||||
suppression: nil)
|
||||
rescue StandardError
|
||||
uploaded_blob&.purge_later
|
||||
raise
|
||||
ensure
|
||||
close_tempfile!
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def source_attributes
|
||||
{ source_kind: @attributes[:source_kind],
|
||||
source_uri: @attributes[:source_uri],
|
||||
source_path: @attributes[:source_path],
|
||||
source_file_id: @attributes[:source_file_id] }
|
||||
end
|
||||
|
||||
def normalized_source_key source
|
||||
MaterialSyncSuppression.normalize_source_key(source_kind: source[:source_kind],
|
||||
source_uri: source[:source_uri],
|
||||
source_path: source[:source_path],
|
||||
source_file_id: source[:source_file_id])
|
||||
end
|
||||
|
||||
def material_attributes_for material
|
||||
attrs =
|
||||
@attributes
|
||||
.except(:source_kind, :source_uri, :source_path, :source_file_id,
|
||||
:file_blob, :file_tempfile, :file_downloader,
|
||||
:filename, :content_type,
|
||||
:file_sha256, :export_path, :profile, :tag, :url)
|
||||
|
||||
if material.new_record?
|
||||
attrs[:tag] = @attributes[:tag] if @attributes.key?(:tag)
|
||||
attrs[:url] = @attributes[:url] if @attributes.key?(:url)
|
||||
return attrs
|
||||
end
|
||||
|
||||
if @attributes.key?(:tag) && @attributes[:tag].present?
|
||||
attrs[:tag] = @attributes[:tag]
|
||||
end
|
||||
if @attributes.key?(:url) && @attributes[:url].present?
|
||||
attrs[:url] = @attributes[:url]
|
||||
end
|
||||
|
||||
attrs
|
||||
end
|
||||
|
||||
def suppression_for source
|
||||
if source[:source_kind] == 'google_drive_file' && source[:source_path].present?
|
||||
return MaterialSyncSuppressionMatcher.match_google_drive_candidate(
|
||||
drive_file_id: source[:source_file_id],
|
||||
relative_path: source[:source_path])
|
||||
end
|
||||
|
||||
MaterialSyncSuppressionMatcher.match(**source)
|
||||
end
|
||||
|
||||
def uploaded_blob!
|
||||
return @attributes[:file_blob] if @attributes[:file_blob]
|
||||
|
||||
tempfile = upload_tempfile
|
||||
return nil unless tempfile
|
||||
|
||||
tempfile.rewind
|
||||
file_sha256 = @attributes[:file_sha256] || Digest::SHA256.file(tempfile.path).hexdigest
|
||||
@attributes[:file_sha256] = file_sha256
|
||||
|
||||
@import_block = MaterialImportBlockMatcher.match_for_sha256(file_sha256)
|
||||
if @import_block
|
||||
tempfile.rewind
|
||||
return nil
|
||||
end
|
||||
|
||||
blob = ActiveStorage::Blob.create_and_upload!(
|
||||
io: tempfile,
|
||||
filename: @attributes[:filename],
|
||||
content_type: @attributes[:content_type])
|
||||
MaterialFileSha256.assign_metadata_sha256!(blob, file_sha256)
|
||||
tempfile.rewind
|
||||
blob
|
||||
end
|
||||
|
||||
def upload_tempfile
|
||||
return @upload_tempfile if defined?(@upload_tempfile)
|
||||
|
||||
@upload_tempfile = @attributes[:file_tempfile]
|
||||
if @upload_tempfile.blank? && @attributes[:file_downloader]
|
||||
@upload_tempfile = @attributes[:file_downloader].call
|
||||
end
|
||||
@upload_tempfile
|
||||
end
|
||||
|
||||
def upsert_export_item! material
|
||||
export_path = resolved_export_path(material)
|
||||
return if export_path.blank?
|
||||
|
||||
item = material.material_export_items.find_or_initialize_by(profile: effective_profile)
|
||||
item.export_path = export_path
|
||||
item.enabled = true
|
||||
item.created_by_user ||= @attributes[:created_by_user]
|
||||
item.save!
|
||||
end
|
||||
|
||||
def export_path_taken_by_other? material, profile, export_path
|
||||
MaterialExportItem
|
||||
.where(profile:, export_path:)
|
||||
.where.not(material_id: material.id)
|
||||
.exists?
|
||||
end
|
||||
|
||||
def close_tempfile!
|
||||
tempfile = upload_tempfile
|
||||
return unless tempfile
|
||||
|
||||
tempfile.close! unless tempfile.closed?
|
||||
rescue StandardError
|
||||
nil
|
||||
end
|
||||
|
||||
def unchanged? material, source
|
||||
return false if material.new_record? || material.discarded?
|
||||
return false unless same_source_attributes?(material, source)
|
||||
return false unless same_export_path?(material)
|
||||
return false unless same_material_attributes?(material)
|
||||
|
||||
same_file_snapshot?(material)
|
||||
end
|
||||
|
||||
def same_source_attributes? material, source
|
||||
material.source_kind == source[:source_kind] \
|
||||
&& material.source_uri == source[:source_uri] \
|
||||
&& material.source_path == source[:source_path] \
|
||||
&& material.source_file_id == source[:source_file_id]
|
||||
end
|
||||
|
||||
def same_export_path? material
|
||||
expected = resolved_export_path(material)
|
||||
current = material.material_export_items.find { |item| item.profile == effective_profile }
|
||||
|
||||
return current.blank? if expected.blank?
|
||||
|
||||
current&.enabled && current.export_path == expected
|
||||
end
|
||||
|
||||
def same_material_attributes? material
|
||||
same_tag_attribute?(material) && same_url_attribute?(material)
|
||||
end
|
||||
|
||||
def same_file_snapshot? material
|
||||
expected_sha256 = @attributes[:file_sha256].to_s.presence
|
||||
|
||||
if expected_sha256.present?
|
||||
return false unless material.file.attached?
|
||||
|
||||
return MaterialFileSha256.metadata_sha256(material.file.blob) == expected_sha256
|
||||
end
|
||||
|
||||
!material.file.attached?
|
||||
end
|
||||
|
||||
def effective_profile
|
||||
@attributes[:profile].presence || 'legacy_drive'
|
||||
end
|
||||
|
||||
def resolved_export_path material
|
||||
export_path = @attributes[:export_path].to_s.strip
|
||||
return export_path if export_path.blank?
|
||||
return export_path unless export_path_taken_by_other?(material, effective_profile, export_path)
|
||||
|
||||
MaterialSyncExportPath.uniquify(export_path, @attributes[:source_file_id])
|
||||
end
|
||||
|
||||
def same_tag_attribute? material
|
||||
return true unless @attributes.key?(:tag)
|
||||
return true if @attributes[:tag].blank?
|
||||
|
||||
material.tag_id == @attributes[:tag].id
|
||||
end
|
||||
|
||||
def same_url_attribute? material
|
||||
return true unless @attributes.key?(:url)
|
||||
return true if @attributes[:url].blank?
|
||||
|
||||
material.url == @attributes[:url]
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,251 @@
|
||||
class MaterialSyncRunner
|
||||
Result = Struct.new(:imported, :updated, :unchanged, :suppressed, :failed,
|
||||
:errors, keyword_init: true)
|
||||
|
||||
def self.sync_enabled!
|
||||
results = MaterialSyncSource.enabled.order(:id).map { |source| new(source).sync! }
|
||||
|
||||
Result.new(imported: results.sum(&:imported),
|
||||
updated: results.sum(&:updated),
|
||||
unchanged: results.sum(&:unchanged),
|
||||
suppressed: results.sum(&:suppressed),
|
||||
failed: results.sum(&:failed),
|
||||
errors: results.flat_map(&:errors))
|
||||
end
|
||||
|
||||
def initialize source
|
||||
@source = source
|
||||
end
|
||||
|
||||
def sync!
|
||||
result = Result.new(imported: 0, updated: 0, unchanged: 0,
|
||||
suppressed: 0, failed: 0, errors: [])
|
||||
|
||||
if @source.source_kind == 'google_drive_path'
|
||||
sync_google_drive_path!(result)
|
||||
else
|
||||
candidates.each do |candidate|
|
||||
next if candidate.blank?
|
||||
|
||||
sync_candidate!(candidate, result)
|
||||
end
|
||||
end
|
||||
|
||||
@source.update!(last_synced_at: Time.current)
|
||||
log_result(result)
|
||||
result
|
||||
rescue NotImplementedError, StandardError => e
|
||||
result.failed += 1
|
||||
result.errors << { source_id: @source.id, error: e.message }
|
||||
log_result(result)
|
||||
result
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def candidates
|
||||
case @source.source_kind
|
||||
when 'uri'
|
||||
[uri_candidate]
|
||||
when 'google_drive_file'
|
||||
[google_drive_file_candidate]
|
||||
when 'legacy_drive_path'
|
||||
raise NotImplementedError, 'legacy_drive_path material sync is not implemented'
|
||||
else
|
||||
raise NotImplementedError, "Unsupported material sync source_kind: #{ @source.source_kind }"
|
||||
end
|
||||
end
|
||||
|
||||
def uri_candidate
|
||||
{ source_kind: @source.source_kind,
|
||||
source_uri: @source.source_uri,
|
||||
source_path: @source.source_path,
|
||||
source_file_id: @source.source_file_id,
|
||||
url: @source.source_uri,
|
||||
tag: nil,
|
||||
created_by_user: @source.created_by_user,
|
||||
updated_by_user: @source.updated_by_user || @source.created_by_user }
|
||||
end
|
||||
|
||||
def sync_candidate! candidate, result
|
||||
block = MaterialImportBlockMatcher.match_for_sha256(candidate[:file_sha256])
|
||||
if block
|
||||
result.suppressed += 1
|
||||
Rails.logger.info(
|
||||
material_sync_log(action: 'suppressed',
|
||||
reason: block.reason,
|
||||
normalized_source_key: candidate_normalized_source_key(candidate)))
|
||||
return
|
||||
end
|
||||
|
||||
suppression = suppression_for(candidate)
|
||||
|
||||
if suppression
|
||||
result.suppressed += 1
|
||||
Rails.logger.info(
|
||||
material_sync_log(action: 'suppressed',
|
||||
normalized_source_key: suppression.normalized_source_key))
|
||||
return
|
||||
end
|
||||
|
||||
import = MaterialSyncImporter.import!(candidate)
|
||||
result.public_send("#{ import.action }=", result.public_send(import.action) + 1)
|
||||
rescue StandardError => e
|
||||
result.failed += 1
|
||||
result.errors << { source_id: @source.id,
|
||||
normalized_source_key: candidate_normalized_source_key(candidate),
|
||||
error: e.message }
|
||||
Rails.logger.warn(
|
||||
material_sync_log(action: 'failed',
|
||||
error: e.message,
|
||||
normalized_source_key: candidate_normalized_source_key(candidate)))
|
||||
ensure
|
||||
close_candidate_tempfile(candidate)
|
||||
end
|
||||
|
||||
def google_drive_path_candidates
|
||||
folder_id = google_drive_folder_id
|
||||
Enumerator.new do |entries|
|
||||
drive_client.each_material_file_under_folder(folder_id).each do |entry|
|
||||
entries << build_google_drive_candidate(entry)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def sync_google_drive_path! result
|
||||
folder_id = google_drive_folder_id
|
||||
scanned_count = 0
|
||||
|
||||
drive_client.each_material_file_under_folder(folder_id) do |entry|
|
||||
scanned_count += 1
|
||||
sync_candidate!(build_google_drive_candidate(entry), result)
|
||||
log_google_drive_progress(folder_id, scanned_count, result) if progress_log_scan_count?(scanned_count)
|
||||
end
|
||||
|
||||
log_google_drive_progress(folder_id, scanned_count, result, summary: true)
|
||||
end
|
||||
|
||||
def google_drive_file_candidate
|
||||
entry = drive_client.fetch_material_file(google_drive_file_id)
|
||||
return nil unless entry
|
||||
|
||||
build_google_drive_candidate(entry, single_file: true)
|
||||
end
|
||||
|
||||
def build_google_drive_candidate entry, single_file: false
|
||||
relative_path =
|
||||
if single_file
|
||||
nil
|
||||
else
|
||||
entry[:relative_path]
|
||||
end
|
||||
export_path =
|
||||
MaterialSyncExportPath.build(prefix: @source.export_path_prefix,
|
||||
relative_path:,
|
||||
filename: entry[:name],
|
||||
source_file_id: entry[:id])
|
||||
|
||||
{ source_kind: 'google_drive_file',
|
||||
source_uri: entry[:web_view_link] || google_drive_file_url(entry[:id]),
|
||||
source_path: relative_path.presence || MaterialSyncExportPath.normalize_path(entry[:name]),
|
||||
source_file_id: entry[:id],
|
||||
filename: entry[:name],
|
||||
content_type: entry[:mime_type],
|
||||
file_downloader: lambda {
|
||||
drive_client.download_to_tempfile(entry[:id], filename: entry[:name])
|
||||
},
|
||||
file_sha256: entry[:sha256_checksum],
|
||||
export_path:,
|
||||
profile: @source.profile,
|
||||
tag: nil,
|
||||
url: nil,
|
||||
created_by_user: @source.created_by_user,
|
||||
updated_by_user: @source.updated_by_user || @source.created_by_user }
|
||||
end
|
||||
|
||||
def candidate_normalized_source_key candidate
|
||||
MaterialSyncSuppression.normalize_source_key(source_kind: candidate[:source_kind],
|
||||
source_uri: candidate[:source_uri],
|
||||
source_path: candidate[:source_path],
|
||||
source_file_id: candidate[:source_file_id])
|
||||
end
|
||||
|
||||
def suppression_for candidate
|
||||
if candidate[:source_kind] == 'google_drive_file' && candidate[:source_path].present?
|
||||
return MaterialSyncSuppressionMatcher.match_google_drive_candidate(
|
||||
drive_file_id: candidate[:source_file_id],
|
||||
relative_path: candidate[:source_path])
|
||||
end
|
||||
|
||||
MaterialSyncSuppressionMatcher.match(source_kind: candidate[:source_kind],
|
||||
source_uri: candidate[:source_uri],
|
||||
source_path: candidate[:source_path],
|
||||
source_file_id: candidate[:source_file_id])
|
||||
end
|
||||
|
||||
def drive_client
|
||||
@drive_client ||= GoogleDrive::ApiClient.new
|
||||
end
|
||||
|
||||
def google_drive_folder_id
|
||||
google_drive_target_id.tap do |id|
|
||||
if id.blank?
|
||||
raise ArgumentError, 'google_drive_path source_file_id or source_uri is required'
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def google_drive_file_id
|
||||
google_drive_target_id.tap do |id|
|
||||
if id.blank?
|
||||
raise ArgumentError, 'google_drive_file source_file_id or source_uri is required'
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def google_drive_target_id
|
||||
@source.source_file_id.presence || drive_client.extract_file_id(@source.source_uri)
|
||||
end
|
||||
|
||||
def google_drive_file_url file_id
|
||||
"https://drive.google.com/file/d/#{ file_id }/view"
|
||||
end
|
||||
|
||||
def close_candidate_tempfile candidate
|
||||
tempfile = candidate[:file_tempfile]
|
||||
return unless tempfile
|
||||
|
||||
tempfile.close! unless tempfile.closed?
|
||||
rescue StandardError
|
||||
nil
|
||||
end
|
||||
|
||||
def log_result result
|
||||
Rails.logger.info(material_sync_log(imported: result.imported,
|
||||
updated: result.updated,
|
||||
unchanged: result.unchanged,
|
||||
suppressed: result.suppressed,
|
||||
failed: result.failed))
|
||||
end
|
||||
|
||||
def progress_log_scan_count? scanned_count
|
||||
scanned_count == 1 || (scanned_count % 50).zero?
|
||||
end
|
||||
|
||||
def log_google_drive_progress folder_id, scanned_count, result, summary: false
|
||||
Rails.logger.info(
|
||||
material_sync_log(folder_id:,
|
||||
scanned_count:,
|
||||
imported: result.imported,
|
||||
updated: result.updated,
|
||||
unchanged: result.unchanged,
|
||||
suppressed: result.suppressed,
|
||||
failed: result.failed,
|
||||
progress: summary ? 'summary' : 'scan'))
|
||||
end
|
||||
|
||||
def material_sync_log fields
|
||||
{ material_sync_source_id: @source.id,
|
||||
material_sync_source_name: @source.name }.merge(fields).to_json
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,68 @@
|
||||
class MaterialSyncSuppressionMatcher
|
||||
def self.match source_kind:,
|
||||
source_uri: nil,
|
||||
drive_path: nil,
|
||||
drive_file_id: nil,
|
||||
source_path: nil,
|
||||
source_file_id: nil
|
||||
key = MaterialSyncSuppression.normalize_source_key(source_kind:,
|
||||
source_uri:,
|
||||
drive_path:,
|
||||
drive_file_id:,
|
||||
source_path:,
|
||||
source_file_id:)
|
||||
return nil if key.blank?
|
||||
|
||||
MaterialSyncSuppression.find_by(normalized_source_key: key)
|
||||
end
|
||||
|
||||
def self.match_google_drive_candidate drive_file_id:, relative_path:
|
||||
normalized_path = validate_relative_path!(relative_path)
|
||||
|
||||
exact = [
|
||||
normalize_key('google_drive_file', drive_file_id),
|
||||
normalize_key('google_drive_path', normalized_path),
|
||||
normalize_key('legacy_drive_path', normalized_path),
|
||||
].compact
|
||||
|
||||
suppression =
|
||||
MaterialSyncSuppression.find_by(normalized_source_key: exact)
|
||||
return suppression if suppression
|
||||
|
||||
MaterialSyncSuppression
|
||||
.where(source_kind: ['google_drive_path_prefix', 'legacy_drive_path_prefix'])
|
||||
.where('drive_path = :path OR :path LIKE CONCAT(drive_path, "/%")', path: normalized_path)
|
||||
.order(:id)
|
||||
.first
|
||||
end
|
||||
|
||||
def self.suppressed?(...)
|
||||
match(...).present?
|
||||
end
|
||||
|
||||
def self.validate_relative_path! relative_path
|
||||
raw = relative_path.to_s.delete("\0").tr('\\', '/').strip
|
||||
raise ArgumentError, 'relative_path is required' if raw.blank?
|
||||
raise ArgumentError, 'relative_path must be relative' if raw.start_with?('/')
|
||||
raise ArgumentError, 'relative_path must be relative' if raw.match?(/\A[A-Za-z]:\//)
|
||||
if raw.start_with?('My Drive/', 'マイドライブ/')
|
||||
raise ArgumentError, 'relative_path must be relative to source folder'
|
||||
end
|
||||
raise ArgumentError, 'relative_path must not contain //' if raw.include?('//')
|
||||
raise ArgumentError, 'relative_path must not end with /' if raw.end_with?('/')
|
||||
|
||||
parts = raw.split('/')
|
||||
if parts.any? { |part| part.in?(['.', '..']) }
|
||||
raise ArgumentError, 'relative_path must not contain dot segments'
|
||||
end
|
||||
|
||||
raw
|
||||
end
|
||||
|
||||
def self.normalize_key source_kind, value
|
||||
return nil if value.blank?
|
||||
|
||||
MaterialSyncSuppression.normalize_source_key(source_kind:, source_path: value,
|
||||
source_file_id: value)
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,51 @@
|
||||
class MaterialSyncSuppressionRegistrar
|
||||
def self.create! attributes, created_by_user:
|
||||
new(attributes, created_by_user:).create!
|
||||
end
|
||||
|
||||
def initialize attributes, created_by_user:
|
||||
@attributes = attributes
|
||||
@created_by_user = created_by_user
|
||||
end
|
||||
|
||||
def create!
|
||||
suppression = nil
|
||||
|
||||
MaterialSyncSuppression.transaction do
|
||||
suppression = MaterialSyncSuppression.create!(
|
||||
@attributes.merge(created_by_user: @created_by_user))
|
||||
discard_existing_materials!(suppression)
|
||||
end
|
||||
|
||||
suppression
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def discard_existing_materials! suppression
|
||||
matching_materials(suppression).find_each do |material|
|
||||
MaterialVersionRecorder.ensure_snapshot!(material, created_by_user: @created_by_user)
|
||||
material.discard!
|
||||
MaterialVersionRecorder.record!(material:,
|
||||
event_type: :discard,
|
||||
created_by_user: @created_by_user)
|
||||
end
|
||||
end
|
||||
|
||||
def matching_materials suppression
|
||||
materials = Material.unscoped.kept
|
||||
|
||||
case suppression.source_kind
|
||||
when 'google_drive_path', 'legacy_drive_path'
|
||||
materials.where(source_path: suppression.drive_path)
|
||||
when 'google_drive_path_prefix', 'legacy_drive_path_prefix'
|
||||
path = suppression.drive_path.to_s
|
||||
materials.where(
|
||||
'source_path = :path OR source_path LIKE :prefix',
|
||||
path:,
|
||||
prefix: "#{ path }/%")
|
||||
else
|
||||
materials.where(normalized_source_key: suppression.normalized_source_key)
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,126 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
require 'mini_magick'
|
||||
require 'open3'
|
||||
require 'tempfile'
|
||||
|
||||
|
||||
class MaterialThumbnailGenerator
|
||||
SIZE = '180x180'
|
||||
|
||||
class << self
|
||||
def generate! material
|
||||
new(material).generate!
|
||||
end
|
||||
end
|
||||
|
||||
def initialize material
|
||||
@material = material
|
||||
@ffmpeg_stderr = []
|
||||
end
|
||||
|
||||
def generate!
|
||||
return log_result(:no_file) unless @material.file.attached?
|
||||
return log_result(:unsupported_content_type) unless image? || video?
|
||||
|
||||
@material.file.blob.open do |file|
|
||||
thumbnail = image? ? image_thumbnail(file.path) : video_thumbnail(file.path)
|
||||
return log_result(:generation_failed) unless thumbnail
|
||||
|
||||
return attach_thumbnail(thumbnail)
|
||||
end
|
||||
rescue ActiveStorage::FileNotFoundError => e
|
||||
log_result(:file_not_found, error: e)
|
||||
rescue MiniMagick::Error => e
|
||||
log_result(:mini_magick_error, error: e)
|
||||
rescue ArgumentError, StandardError => e
|
||||
log_result(:generation_failed, error: e)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def image? = content_type.start_with?('image/')
|
||||
|
||||
def video? = content_type.start_with?('video/')
|
||||
|
||||
def content_type
|
||||
return nil unless @material.file.attached?
|
||||
|
||||
@material.file.blob.content_type.to_s
|
||||
end
|
||||
|
||||
def image_thumbnail path
|
||||
image = MiniMagick::Image.open(path)
|
||||
image.resize(SIZE)
|
||||
image.format('jpg')
|
||||
image
|
||||
end
|
||||
|
||||
def video_thumbnail path
|
||||
[1, 0].each do |seconds|
|
||||
tempfile = Tempfile.new(['material-thumbnail', '.jpg'])
|
||||
tempfile.close
|
||||
ok = extract_video_frame(path, tempfile.path, seconds)
|
||||
next unless ok && File.size?(tempfile.path)
|
||||
|
||||
return image_thumbnail(tempfile.path)
|
||||
ensure
|
||||
tempfile&.unlink
|
||||
end
|
||||
|
||||
nil
|
||||
end
|
||||
|
||||
def extract_video_frame input_path, output_path, seconds
|
||||
_stdout, stderr, status =
|
||||
Open3.capture3('ffmpeg',
|
||||
'-y',
|
||||
'-ss', seconds.to_s,
|
||||
'-i', input_path,
|
||||
'-frames:v', '1',
|
||||
'-f', 'image2',
|
||||
output_path)
|
||||
@ffmpeg_stderr << stderr if stderr.present?
|
||||
status.success?
|
||||
rescue Errno::ENOENT => e
|
||||
@ffmpeg_stderr << "ffmpeg unavailable: #{ e.message }"
|
||||
false
|
||||
end
|
||||
|
||||
def attach_thumbnail image
|
||||
blob = nil
|
||||
File.open(image.path) do |io|
|
||||
blob = ActiveStorage::Blob.create_and_upload!(
|
||||
io:,
|
||||
filename: 'material-thumbnail.jpg',
|
||||
content_type: 'image/jpeg')
|
||||
end
|
||||
@material.thumbnail.attach(blob)
|
||||
log_result(:attached)
|
||||
rescue StandardError => e
|
||||
blob&.purge_later
|
||||
log_result(:attach_failed, error: e)
|
||||
end
|
||||
|
||||
def log_result result, error: nil
|
||||
log_payload = { material_id: @material.id,
|
||||
file_blob_id: file_blob_id,
|
||||
content_type:,
|
||||
result:,
|
||||
error_class: error&.class&.name,
|
||||
error_message: error&.message,
|
||||
ffmpeg_stderr: @ffmpeg_stderr.join("\n").presence }
|
||||
if [:attached, :no_file, :unsupported_content_type].include?(result)
|
||||
Rails.logger.info("Material thumbnail generation: #{ log_payload.to_json }")
|
||||
else
|
||||
Rails.logger.warn("Material thumbnail generation: #{ log_payload.to_json }")
|
||||
end
|
||||
result
|
||||
end
|
||||
|
||||
def file_blob_id
|
||||
return nil unless @material.file.attached?
|
||||
|
||||
@material.file.blob.id
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,73 @@
|
||||
class MaterialVersionRecorder < VersionRecorder
|
||||
EVENT_TYPES = ['create', 'update', 'discard', 'restore'].freeze
|
||||
|
||||
def self.record! material:, event_type:, created_by_user:, file_snapshot: nil
|
||||
new(material:, event_type:, created_by_user:, file_snapshot:).record!
|
||||
end
|
||||
|
||||
def initialize material:, event_type:, created_by_user:, file_snapshot: nil
|
||||
@file_snapshot = file_snapshot
|
||||
|
||||
super(record: material, event_type:, created_by_user:)
|
||||
end
|
||||
|
||||
def self.ensure_snapshot! material, created_by_user:
|
||||
return if material.material_versions.exists?
|
||||
|
||||
record!(material:, event_type: :create,
|
||||
created_by_user: material.created_by_user || created_by_user)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def version_class = MaterialVersion
|
||||
def version_association = :material_versions
|
||||
def record_key = :material
|
||||
|
||||
def snapshot_attributes
|
||||
blob = @record.file.attached? ? @record.file.blob : nil
|
||||
file_snapshot = build_file_snapshot(blob)
|
||||
|
||||
{ url: @record.url,
|
||||
parent: @record.parent,
|
||||
tag: @record.tag,
|
||||
tag_name: @record.tag&.name,
|
||||
tag_category: @record.tag&.category,
|
||||
source_kind: @record.source_kind,
|
||||
source_uri: @record.source_uri,
|
||||
source_path: @record.source_path,
|
||||
source_file_id: @record.source_file_id,
|
||||
normalized_source_key: @record.normalized_source_key,
|
||||
export_paths_json: @record.snapshot_export_paths,
|
||||
discarded_at: @record.discarded_at,
|
||||
file_blob_id: file_snapshot[:file_blob_id],
|
||||
file_filename: file_snapshot[:file_filename],
|
||||
file_content_type: file_snapshot[:file_content_type],
|
||||
file_byte_size: file_snapshot[:file_byte_size],
|
||||
file_checksum: file_snapshot[:file_checksum],
|
||||
file_sha256: file_snapshot[:file_sha256] }
|
||||
end
|
||||
|
||||
def build_file_snapshot blob
|
||||
return @file_snapshot if @file_snapshot
|
||||
return empty_file_snapshot unless blob
|
||||
|
||||
{ file_blob_id: blob.id,
|
||||
file_filename: blob.filename.to_s,
|
||||
file_content_type: blob.content_type,
|
||||
file_byte_size: blob.byte_size,
|
||||
file_checksum: blob.checksum,
|
||||
file_sha256: MaterialFileSha256.metadata_sha256(blob) }
|
||||
end
|
||||
|
||||
def empty_file_snapshot
|
||||
{ file_blob_id: nil,
|
||||
file_filename: nil,
|
||||
file_content_type: nil,
|
||||
file_byte_size: nil,
|
||||
file_checksum: nil,
|
||||
file_sha256: nil }
|
||||
end
|
||||
|
||||
def event_types = self.class::EVENT_TYPES
|
||||
end
|
||||
@@ -0,0 +1,148 @@
|
||||
require 'stringio'
|
||||
require 'zlib'
|
||||
|
||||
# Initial implementation keeps every file payload and the final ZIP in memory.
|
||||
# Keep this service boundary stable so job/cached export paths can replace it later.
|
||||
class MaterialZipExporter
|
||||
Entry = Struct.new(:path, :data, :mtime, keyword_init: true)
|
||||
MissingFile = Struct.new(:material_id, :export_path, :blob_id, :filename, keyword_init: true)
|
||||
|
||||
class EmptyExportError < StandardError; end
|
||||
class DuplicatePathError < StandardError; end
|
||||
class MissingFileError < StandardError
|
||||
attr_reader :missing_files
|
||||
|
||||
def initialize missing_files
|
||||
@missing_files = missing_files
|
||||
|
||||
super("Missing files: #{missing_files.map(&:export_path).join(', ')}")
|
||||
end
|
||||
end
|
||||
|
||||
def initialize profile: 'legacy_drive', tag_id: nil
|
||||
@profile = profile.presence || 'legacy_drive'
|
||||
@tag_id = tag_id.presence
|
||||
end
|
||||
|
||||
def export
|
||||
entries = build_entries
|
||||
raise EmptyExportError if entries.empty?
|
||||
|
||||
ZipWriter.write(entries)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def build_entries
|
||||
rows = MaterialExportItem
|
||||
.enabled
|
||||
.includes(material: { file_attachment: :blob })
|
||||
.joins(:material)
|
||||
.merge(Material.kept)
|
||||
.where(profile: @profile)
|
||||
.order(:export_path)
|
||||
|
||||
rows = rows.where(materials: { tag_id: @tag_id }) if @tag_id
|
||||
|
||||
missing_files = []
|
||||
|
||||
entries = rows.filter_map do |item|
|
||||
material = item.material
|
||||
next unless material.file.attached?
|
||||
|
||||
data = download_blob(item, missing_files)
|
||||
next unless data
|
||||
|
||||
Entry.new(path: item.export_path,
|
||||
data:,
|
||||
mtime: material.updated_at || Time.current)
|
||||
end
|
||||
|
||||
raise MissingFileError.new(missing_files) if missing_files.any?
|
||||
|
||||
paths = entries.map(&:path)
|
||||
duplicated = paths.find { |path| paths.count(path) > 1 }
|
||||
raise DuplicatePathError, duplicated if duplicated
|
||||
|
||||
entries
|
||||
end
|
||||
|
||||
def download_blob item, missing_files
|
||||
blob = item.material.file.blob
|
||||
|
||||
blob.download
|
||||
rescue ActiveStorage::FileNotFoundError
|
||||
missing_files << MissingFile.new(
|
||||
material_id: item.material_id,
|
||||
export_path: item.export_path,
|
||||
blob_id: blob.id,
|
||||
filename: blob.filename.to_s,
|
||||
)
|
||||
nil
|
||||
end
|
||||
|
||||
class ZipWriter
|
||||
VERSION_NEEDED = 20
|
||||
GP_FLAG = 0x0800
|
||||
COMPRESSION_STORE = 0
|
||||
|
||||
def self.write entries
|
||||
new(entries).write
|
||||
end
|
||||
|
||||
def initialize entries
|
||||
@entries = entries
|
||||
@central_directory = []
|
||||
end
|
||||
|
||||
def write
|
||||
io = StringIO.new(''.b)
|
||||
|
||||
@entries.each do |entry|
|
||||
write_entry(io, entry)
|
||||
end
|
||||
|
||||
central_start = io.pos
|
||||
@central_directory.each { |header| io.write(header) }
|
||||
central_size = io.pos - central_start
|
||||
|
||||
io.write([0x06054b50, 0, 0, @entries.size, @entries.size,
|
||||
central_size, central_start, 0].pack('VvvvvVVv'))
|
||||
io.string
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def write_entry io, entry
|
||||
path = entry.path.b
|
||||
data = entry.data.b
|
||||
crc32 = Zlib.crc32(data)
|
||||
dos_time, dos_date = dos_timestamp(entry.mtime)
|
||||
offset = io.pos
|
||||
|
||||
local_header = [0x04034b50, VERSION_NEEDED, GP_FLAG, COMPRESSION_STORE,
|
||||
dos_time, dos_date, crc32, data.bytesize, data.bytesize,
|
||||
path.bytesize, 0].pack('VvvvvvVVVvv')
|
||||
|
||||
io.write(local_header)
|
||||
io.write(path)
|
||||
io.write(data)
|
||||
|
||||
@central_directory << central_header(path:, crc32:, size: data.bytesize,
|
||||
dos_time:, dos_date:, offset:)
|
||||
end
|
||||
|
||||
def central_header path:, crc32:, size:, dos_time:, dos_date:, offset:
|
||||
[0x02014b50, VERSION_NEEDED, VERSION_NEEDED, GP_FLAG, COMPRESSION_STORE,
|
||||
dos_time, dos_date, crc32, size, size, path.bytesize, 0, 0, 0, 0, 0,
|
||||
offset].pack('VvvvvvvVVVvvvvvVV') + path
|
||||
end
|
||||
|
||||
def dos_timestamp time
|
||||
local = time.to_time
|
||||
dos_time = (local.hour << 11) | (local.min << 5) | (local.sec / 2)
|
||||
dos_date = ((local.year - 1980) << 9) | (local.month << 5) | local.day
|
||||
[dos_time, dos_date]
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -73,7 +73,7 @@ class VersionRecorder
|
||||
end
|
||||
|
||||
def validate_event_type!
|
||||
return if EVENT_TYPES.include?(@event_type)
|
||||
return if event_types.include?(@event_type)
|
||||
|
||||
raise ArgumentError, "Invalid event_type: #{ @event_type }"
|
||||
end
|
||||
@@ -84,4 +84,5 @@ class VersionRecorder
|
||||
def snapshot_attributes = raise NotImplementedError
|
||||
|
||||
def record_class = @record.class
|
||||
def event_types = self.class::EVENT_TYPES
|
||||
end
|
||||
|
||||
新しい課題から参照
ユーザをブロックする