開発環境では、**DB を壊さない前提で、migration → API → 画面 → 同期 → ZIP → 履歴**の順に見るのがよいです。今回の差分は素材管理全体に触っているので、単体でチョンチョン見るより、素材の一生を通すのが早いです。
## 0. 先に方針
**やらないこと:**
```sh id="snb3i6"
rails db:drop
rails db:reset
rails db:setup
DISABLE_DATABASE_ENVIRONMENT_CHECK=1 ...
```
これは禁止。
開発 DB に本番データを入れているなら、床板を剥がして耐震確認するようなものです。
---
## 1. migration 確認
まず現在の状態を見る。
```sh id="qsdfpt"
cd backend
RAILS_ENV=development bundle exec rails db:migrate:status
```
その後、通常 migration。
```sh id="a89fir"
RAILS_ENV=development bundle exec rails db:migrate
```
見るポイント:
```txt id="c1u57a"
materials に source_* / normalized_source_key / version_no がある
material_versions に event_type / file snapshot / source snapshot がある
material_export_items がある
material_sync_suppressions がある
material_sync_sources がある
既存 materials に material_versions version_no=1 create が backfill されている
```
確認用:
```sh id="g4vd4m"
RAILS_ENV=development bundle exec rails runner '
puts "materials=#{Material.count}"
puts "versions=#{MaterialVersion.count}"
puts "materials without versions=#{Material.left_joins(:material_versions).where(material_versions: { id: nil }).count}"
puts "sync suppressions table=#{ActiveRecord::Base.connection.table_exists?(:material_sync_suppressions)}"
'
```
ここで `materials without versions=0` になれば、backfill は通っています。
---
## 2. 既存素材一覧の画面確認
フロントを起動して `/materials` を見る。
```sh id="vl28jd"
cd frontend
npm run dev
```
見る観点:
```txt id="i2ovxw"
初期表示で素材が出る
初期表示ではグルーピングがオフ
タグなし素材も出る
カード表示でサムネまたは代替テキストが出る
一覧表示に切り替えられる
q / tag_state / media_kind / sort / direction が効く
```
ここでまず、普通の素材一覧が壊れていないことを確認します。
---
## 3. 左タグバーの確認
`/materials` を開いて左タグバーからタグを選ぶ。
見る観点:
```txt id="n6udul"
URL が tag_id=...&include_descendants=1&group_by=parent_tag になる
選択中タグが左バーで強調される
一覧上部に「選択中」の表示が出る
「タグ選択を解除」で通常表示に戻れる
解除後、tag_id / include_descendants / group_by / page が消える
子タグ・孫タグの素材も一覧に出る
親タググルーピングされる
```
特に重要なのはこれ。
```txt id="d33os0"
親タグ A を選択
A に直接紐づく素材
A > B に紐づく素材
A > B > C に紐づく素材
が同じ一覧に出ること
```
---
## 4. 選択タグから素材追加
左タグからタグを選択した状態で、一覧上部の **このタグに素材を追加** を押す。
見る観点:
```txt id="qdd5uw"
素材追加画面の tag 欄に選択中タグ名が初期入力されている
file または url を指定して保存できる
保存後 return_to で元のタグ選択済み一覧に戻る
戻った一覧に追加した素材が出る
material_versions に create が 1 件できる
```
Rails console でも確認できます。
```sh id="i07rrb"
RAILS_ENV=development bundle exec rails runner '
m = Material.order(id: :desc).first
puts({ id: m.id, tag: m.tag&.name, versions: m.material_versions.count, version_no: m.version_no }.inspect)
'
```
---
## 5. グループ見出しから素材追加
親タググルーピング表示中に、各グループ見出しの **このタグに素材を追加** を押す。
見る観点:
```txt id="fpx8w4"
グループタグ名が tag 欄に初期入力される
保存後、元の一覧に戻る
追加素材がそのグループ内に出る
```
ここは今回の導線の肝です。棚の見出しから直接その棚へ素材を置けるかを見る。
---
## 6. 素材更新と履歴
既存素材の詳細または編集導線から、タグ・URL・ファイル・export path を更新する。
見る観点:
```txt id="w36i61"
更新前 snapshot が無ければ create が補われる
更新後 update version ができる
file_blob_id / file_filename / file_sha256 が material_versions に入る
export_paths_json が履歴に残る
/materials/changes または /materials/versions で履歴が見える
```
console:
```sh id="a3okhv"
RAILS_ENV=development bundle exec rails runner '
m = Material.order(updated_at: :desc).first
puts m.material_versions.order(:version_no).map { |v|
[v.version_no, v.event_type, v.tag_name, v.file_filename, v.file_sha256, v.export_paths_hash]
}.inspect
'
```
---
## 7. サムネイル
画像素材を追加して、一覧にサムネイルが出るか確認。
動画素材があるなら、`ffmpeg` が入っている環境で backfill。
```sh id="qxm8fj"
cd backend
RAILS_ENV=development bundle exec rails materials:thumbnails:backfill
```
見る観点:
```txt id="s9a1vf"
画像は 180x180 のサムネが付く
動画はフレームからサムネが作られる
非対応ファイルは代替テキスト表示になる
ログに result が出る
```
---
## 8. ZIP export
export path がある素材を用意して、ブラウザで確認。
```txt id="aq7f7o"
/materials/download.zip?profile=legacy_drive
```
見る観点:
```txt id="h4rve5"
ZIP が落ちる
entry path が material_export_items.export_path になる
disabled な export item は入らない
ファイル実体が欠けている場合は 422 と missing_files が返る
```
---
## 9. 抑止
`/materials/suppressions` で path prefix 抑止を追加する。
見る観点:
```txt id="w80jbu"
member で作成できる
guest は forbidden / unauthorized
google_drive_path_prefix で既存素材が discard される
discard 履歴が material_versions に残る
同期時に同じ source_path 配下が再作成されない
```
---
## 10. Google Drive 同期
開発環境では、まず小さいフォルダでやるのがよいです。
いきなり本番素材集フォルダを食わせると、ログが藪になります 🌿
必要な ENV:
```sh id="kzbb7f"
GOOGLE_DRIVE_SERVICE_ACCOUNT_EMAIL=...
GOOGLE_DRIVE_PRIVATE_KEY_PATH=...
MATERIAL_SYNC_SOURCE_KIND=google_drive_path
MATERIAL_SYNC_SOURCE_FILE_ID=<folder_id>
MATERIAL_SYNC_SOURCE_NAME=dev-small-folder
MATERIAL_SYNC_SOURCE_PROFILE=legacy_drive
```
seed で source 作成、または console で作成。
```sh id="sec40c"
RAILS_ENV=development bundle exec rails db:seed
RAILS_ENV=development bundle exec rails materials:sync
```
見る観点:
```txt id="a5ltqu"
imported / updated / unchanged / suppressed / failed がログに出る
2 回目実行で unchanged が増える
tag は nil のままでも保存できる
人手で tag / url を付けた既存同期素材が、再同期で消えない
Google native file は skip される
download 後 sha256 block が効く
```
---
## 11. schema.rb は別途確認
これはテストというより merge gate です。
今回まだ怪しいので、差分に以下が混ざっていないことを確認します。
```txt id="s3k6da"
wiki_assets 削除
wiki_pages.next_asset_no 削除
素材管理と無関係な CHECK constraint 削除
素材管理と無関係な index order 消失
```
ここが残るなら、機能テストが通っても merge は止めた方がいいです。
---
## 最小テスト順
時間がないなら、この順で十分です。
```txt id="c1k2pw"
1. db:migrate
2. materials without versions = 0 を確認
3. /materials 初期表示
4. 左タグ選択 → 子孫込み表示 → グルーピング
5. タグ選択解除
6. 選択タグから素材追加 → return_to で戻る
7. グループ見出しから素材追加
8. 更新して material_versions を確認
9. ZIP export
10. 小さい Drive folder で materials:sync を 2 回
```
これで、今回の差分の主要な導線はほぼ踏めます。
Reviewed-on: #381
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
このコミットはPull リクエスト #381 でマージされました．
このコミットが含まれているのは:
@@ -0,0 +1,49 @@
|
||||
class MaterialSyncSuppressionsController < ApplicationController
|
||||
def index
|
||||
return head :unauthorized unless current_user
|
||||
return head :forbidden unless current_user.gte_member?
|
||||
|
||||
suppressions =
|
||||
MaterialSyncSuppression
|
||||
.includes(:created_by_user)
|
||||
.order(created_at: :desc, id: :desc)
|
||||
|
||||
render json: { suppressions: suppressions.map { |s| repr(s) } }
|
||||
end
|
||||
|
||||
def create
|
||||
return head :unauthorized unless current_user
|
||||
return head :forbidden unless current_user.gte_member?
|
||||
|
||||
suppression =
|
||||
MaterialSyncSuppressionRegistrar.create!(suppression_params,
|
||||
created_by_user: current_user)
|
||||
|
||||
render json: repr(suppression), status: :created
|
||||
rescue ActiveRecord::RecordInvalid => e
|
||||
render_validation_error e.record
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def suppression_params
|
||||
params.permit(:source_kind,
|
||||
:source_uri,
|
||||
:drive_path,
|
||||
:drive_file_id,
|
||||
:reason)
|
||||
end
|
||||
|
||||
def repr suppression
|
||||
{ id: suppression.id,
|
||||
source_kind: suppression.source_kind,
|
||||
source_uri: suppression.source_uri,
|
||||
drive_path: suppression.drive_path,
|
||||
drive_file_id: suppression.drive_file_id,
|
||||
normalized_source_key: suppression.normalized_source_key,
|
||||
reason: suppression.reason,
|
||||
created_by_user: suppression.created_by_user&.as_json(UserRepr::BASE),
|
||||
created_at: suppression.created_at,
|
||||
updated_at: suppression.updated_at }
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,54 @@
|
||||
class MaterialVersionsController < ApplicationController
|
||||
def index
|
||||
page = (params[:page].presence || 1).to_i
|
||||
limit = (params[:limit].presence || 20).to_i
|
||||
|
||||
page = 1 if page < 1
|
||||
limit = 1 if limit < 1
|
||||
|
||||
offset = (page - 1) * limit
|
||||
|
||||
q = MaterialVersion.includes(:created_by_user)
|
||||
q = q.where(material_id: params[:material_id]) if params[:material_id].present?
|
||||
q = q.where(tag_name: params[:tag].to_s.strip) if params[:tag].present?
|
||||
q = q.where(event_type: params[:event_type]) if params[:event_type].present?
|
||||
|
||||
count = q.except(:order, :limit, :offset).count
|
||||
versions =
|
||||
q
|
||||
.order(Arel.sql('material_versions.created_at DESC, material_versions.id DESC'))
|
||||
.limit(limit)
|
||||
.offset(offset)
|
||||
|
||||
render json: { versions: serialise_versions(versions), count: }
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def serialise_versions rows
|
||||
rows.map do |row|
|
||||
{ id: row.id,
|
||||
material_id: row.material_id,
|
||||
version_no: row.version_no,
|
||||
event_type: row.event_type,
|
||||
tag_id: row.tag_id,
|
||||
tag_name: row.tag_name,
|
||||
tag_category: row.tag_category,
|
||||
url: row.url,
|
||||
source_kind: row.source_kind,
|
||||
source_uri: row.source_uri,
|
||||
source_path: row.source_path,
|
||||
source_file_id: row.source_file_id,
|
||||
normalized_source_key: row.normalized_source_key,
|
||||
file_blob_id: row.file_blob_id,
|
||||
file_filename: row.file_filename,
|
||||
file_content_type: row.file_content_type,
|
||||
file_byte_size: row.file_byte_size,
|
||||
file_sha256: row.file_sha256,
|
||||
export_paths_json: row.export_paths_hash,
|
||||
discarded_at: row.discarded_at,
|
||||
created_by_user: row.created_by_user&.as_json(UserRepr::BASE),
|
||||
created_at: row.created_at }
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -1,4 +1,11 @@
|
||||
require 'set'
|
||||
|
||||
|
||||
class MaterialsController < ApplicationController
|
||||
rescue_from MaterialZipExporter::EmptyExportError, with: :render_zip_empty
|
||||
rescue_from MaterialZipExporter::DuplicatePathError, with: :render_zip_duplicate_path
|
||||
rescue_from MaterialZipExporter::MissingFileError, with: :render_zip_missing_file
|
||||
|
||||
def index
|
||||
page = (params[:page].presence || 1).to_i
|
||||
limit = (params[:limit].presence || 20).to_i
|
||||
@@ -8,54 +15,100 @@ class MaterialsController < ApplicationController
|
||||
|
||||
offset = (page - 1) * limit
|
||||
|
||||
tag_id = params[:tag_id].presence
|
||||
parent_id = params[:parent_id].presence
|
||||
filters = material_index_filters
|
||||
tag_graph = material_index_tag_graph(filters)
|
||||
q = Material.includes(:material_export_items,
|
||||
thumbnail_attachment: :blob,
|
||||
file_attachment: :blob,
|
||||
tag: :tag_name)
|
||||
q = q.where(tag_id: nil) if filters[:tag_state] == 'untagged'
|
||||
q = q.where.not(tag_id: nil) if filters[:tag_state] == 'tagged'
|
||||
q = apply_material_tag_filter(q, filters, tag_graph)
|
||||
q = q.where('materials.created_at >= ?', filters[:created_from]) if filters[:created_from]
|
||||
q = q.where('materials.created_at <= ?', filters[:created_to]) if filters[:created_to]
|
||||
q = q.where('materials.updated_at >= ?', filters[:updated_from]) if filters[:updated_from]
|
||||
q = q.where('materials.updated_at <= ?', filters[:updated_to]) if filters[:updated_to]
|
||||
|
||||
q = Material.includes(:tag, :created_by_user).with_attached_file
|
||||
q = q.where(tag_id:) if tag_id
|
||||
q = q.where(parent_id:) if parent_id
|
||||
q = material_index_join_tag_name(q) if material_index_needs_tag_name?(filters)
|
||||
q = material_index_join_file_blob(q) if material_index_needs_file_blob?(filters)
|
||||
q = apply_material_query(q, filters[:q]) if filters[:q].present?
|
||||
q = apply_material_media_kind(q, filters[:media_kind])
|
||||
|
||||
count = q.count
|
||||
materials = q.order(created_at: :desc, id: :desc).limit(limit).offset(offset)
|
||||
count = q.distinct.count(:id)
|
||||
materials =
|
||||
q
|
||||
.order(Arel.sql(material_index_order_sql(filters)))
|
||||
.limit(limit)
|
||||
.offset(offset)
|
||||
.to_a
|
||||
|
||||
render json: { materials: MaterialRepr.many(materials, host: request.base_url), count: count }
|
||||
response = { materials: MaterialRepr.list_many(materials, host: request.base_url),
|
||||
count: }
|
||||
if filters[:tag_id]
|
||||
response[:tag_scope] = material_index_tag_scope(filters, tag_graph)
|
||||
end
|
||||
if filters[:group_by] == 'parent_tag'
|
||||
response[:groups] = material_index_groups(materials, tag_graph)
|
||||
end
|
||||
|
||||
render json: response
|
||||
end
|
||||
|
||||
def show
|
||||
material =
|
||||
Material
|
||||
.includes(:tag)
|
||||
.includes(:tag, :material_export_items)
|
||||
.with_attached_thumbnail
|
||||
.with_attached_file
|
||||
.find_by(id: params[:id])
|
||||
return head :not_found unless material
|
||||
|
||||
wiki_page_body = material.tag.tag_name.wiki_page&.current_revision&.body
|
||||
wiki_page_body = material.tag&.tag_name&.wiki_page&.current_revision&.body
|
||||
|
||||
render json: MaterialRepr.base(material, host: request.base_url).merge(wiki_page_body:)
|
||||
end
|
||||
|
||||
def create
|
||||
return head :unauthorized unless current_user
|
||||
return head :forbidden unless current_user.gte_member?
|
||||
|
||||
tag_name_raw = params[:tag].to_s.strip
|
||||
file = params[:file]
|
||||
file_sha256 = MaterialFileSha256.from_upload(file)
|
||||
url = params[:url].to_s.strip.presence
|
||||
return render_unprocessable_entity('タグは必須です．', field: :tag) if tag_name_raw.blank?
|
||||
if file.blank? && url.blank?
|
||||
return render_validation_error fields: { file: ['ファイルまたは URL は必須です．'],
|
||||
url: ['ファイルまたは URL は必須です．'] }
|
||||
end
|
||||
block = MaterialImportBlockMatcher.match_for_sha256(file_sha256)
|
||||
return render_material_import_block(block) if block
|
||||
|
||||
tag_name = TagName.find_undiscard_or_create_by!(name: tag_name_raw)
|
||||
tag = tag_name.tag
|
||||
tag = Tag.create!(tag_name:, category: :material) unless tag
|
||||
uploaded_blob = build_uploaded_material_blob!(file, file_sha256)
|
||||
material = nil
|
||||
|
||||
material = Material.new(tag:, url:,
|
||||
created_by_user: current_user,
|
||||
updated_by_user: current_user)
|
||||
material.file.attach(file)
|
||||
begin
|
||||
Material.transaction do
|
||||
tag_name = TagName.find_undiscard_or_create_by!(name: tag_name_raw)
|
||||
tag = tag_name.tag
|
||||
tag = Tag.create!(tag_name:, category: :material) unless tag
|
||||
|
||||
if material.save
|
||||
material = Material.new(tag:, url:,
|
||||
created_by_user: current_user,
|
||||
updated_by_user: current_user)
|
||||
material.file.attach(uploaded_blob) if uploaded_blob
|
||||
material.save!
|
||||
upsert_export_paths!(material)
|
||||
MaterialVersionRecorder.record!(material:, event_type: :create,
|
||||
created_by_user: current_user)
|
||||
end
|
||||
rescue StandardError
|
||||
uploaded_blob&.purge_later
|
||||
raise
|
||||
end
|
||||
|
||||
if material
|
||||
log_thumbnail_generation(material, MaterialThumbnailGenerator.generate!(material))
|
||||
render json: MaterialRepr.base(material, host: request.base_url), status: :created
|
||||
else
|
||||
render_validation_error material
|
||||
@@ -71,29 +124,44 @@ class MaterialsController < ApplicationController
|
||||
|
||||
tag_name_raw = params[:tag].to_s.strip
|
||||
file = params[:file]
|
||||
url = params[:url].to_s.strip.presence
|
||||
file_sha256 = MaterialFileSha256.from_upload(file)
|
||||
url = params.key?(:url) ? params[:url].to_s.strip.presence : material.url
|
||||
return render_unprocessable_entity('タグは必須です．', field: :tag) if tag_name_raw.blank?
|
||||
if file.blank? && url.blank?
|
||||
if file.blank? && url.blank? && !material.file.attached?
|
||||
return render_validation_error fields: { file: ['ファイルまたは URL は必須です．'],
|
||||
url: ['ファイルまたは URL は必須です．'] }
|
||||
end
|
||||
block = MaterialImportBlockMatcher.match_for_sha256(file_sha256)
|
||||
return render_material_import_block(block) if block
|
||||
|
||||
tag_name = TagName.find_undiscard_or_create_by!(name: tag_name_raw)
|
||||
tag = tag_name.tag
|
||||
tag = Tag.create!(tag_name:, category: :material) unless tag
|
||||
uploaded_blob = build_uploaded_material_blob!(file, file_sha256)
|
||||
|
||||
material.update!(tag:, url:, updated_by_user: current_user)
|
||||
if file
|
||||
material.file.attach(file)
|
||||
else
|
||||
material.file.purge
|
||||
begin
|
||||
Material.transaction do
|
||||
MaterialVersionRecorder.ensure_snapshot!(material, created_by_user: current_user)
|
||||
tag_name = TagName.find_undiscard_or_create_by!(name: tag_name_raw)
|
||||
tag = tag_name.tag
|
||||
tag = Tag.create!(tag_name:, category: :material) unless tag
|
||||
|
||||
material.assign_attributes(tag:, url:, updated_by_user: current_user)
|
||||
if uploaded_blob
|
||||
material.file.attach(uploaded_blob)
|
||||
elsif params.key?(:url) && url.present? && file.blank?
|
||||
material.file.detach
|
||||
end
|
||||
material.save!
|
||||
upsert_export_paths!(material)
|
||||
MaterialVersionRecorder.record!(material:, event_type: :update,
|
||||
created_by_user: current_user)
|
||||
end
|
||||
rescue StandardError
|
||||
uploaded_blob&.purge_later
|
||||
raise
|
||||
end
|
||||
|
||||
if material.save
|
||||
render json: MaterialRepr.base(material, host: request.base_url)
|
||||
else
|
||||
render_validation_error material
|
||||
end
|
||||
log_thumbnail_generation(material, MaterialThumbnailGenerator.generate!(material))
|
||||
|
||||
render json: MaterialRepr.base(material, host: request.base_url)
|
||||
end
|
||||
|
||||
def destroy
|
||||
@@ -103,7 +171,375 @@ class MaterialsController < ApplicationController
|
||||
material = Material.find_by(id: params[:id])
|
||||
return head :not_found unless material
|
||||
|
||||
material.discard
|
||||
Material.transaction do
|
||||
MaterialVersionRecorder.ensure_snapshot!(material, created_by_user: current_user)
|
||||
material.discard!
|
||||
MaterialVersionRecorder.record!(material:, event_type: :discard,
|
||||
created_by_user: current_user)
|
||||
end
|
||||
head :no_content
|
||||
end
|
||||
|
||||
def download
|
||||
zip = MaterialZipExporter.new(profile: params[:profile],
|
||||
tag_id: params[:tag_id]).export
|
||||
profile = params[:profile].presence || 'legacy_drive'
|
||||
send_data zip,
|
||||
type: 'application/zip',
|
||||
disposition: 'attachment',
|
||||
filename: "btrc-materials-#{ profile }.zip"
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def material_index_filters
|
||||
tag_state = params[:tag_state].to_s.presence
|
||||
tag_state = 'untagged' if bool?(:unclassified)
|
||||
tag_state = 'all' unless ['all', 'tagged', 'untagged'].include?(tag_state)
|
||||
|
||||
media_kind = params[:media_kind].to_s.presence
|
||||
unless ['all', 'image', 'video', 'audio', 'file_other', 'url_only'].include?(media_kind)
|
||||
media_kind = 'all'
|
||||
end
|
||||
|
||||
sort = params[:sort].to_s.presence
|
||||
unless ['created_at', 'updated_at', 'tag_name', 'media_kind', 'file_byte_size',
|
||||
'version_no', 'id'].include?(sort)
|
||||
sort = 'created_at'
|
||||
end
|
||||
|
||||
direction = params[:direction].to_s.downcase
|
||||
direction = 'desc' unless ['asc', 'desc'].include?(direction)
|
||||
|
||||
group_by = params[:group_by].to_s.presence
|
||||
group_by = 'none' unless ['none', 'parent_tag'].include?(group_by)
|
||||
|
||||
tag_id = params[:tag_id].to_i
|
||||
tag_id = nil if tag_id <= 0
|
||||
|
||||
{ q: params[:q].to_s.strip.presence,
|
||||
tag_state:,
|
||||
media_kind:,
|
||||
tag_id:,
|
||||
include_descendants: bool?(:include_descendants),
|
||||
group_by:,
|
||||
created_from: parse_time_param(:created_from),
|
||||
created_to: parse_time_param(:created_to),
|
||||
updated_from: parse_time_param(:updated_from),
|
||||
updated_to: parse_time_param(:updated_to),
|
||||
sort:,
|
||||
direction: }
|
||||
end
|
||||
|
||||
def parse_time_param name
|
||||
value = params[name].to_s.strip
|
||||
return nil if value.blank?
|
||||
|
||||
Time.zone.parse(value)
|
||||
rescue ArgumentError
|
||||
nil
|
||||
end
|
||||
|
||||
def material_index_needs_tag_name? filters
|
||||
filters[:q].present? || filters[:sort] == 'tag_name'
|
||||
end
|
||||
|
||||
def material_index_needs_file_blob? filters
|
||||
filters[:q].present? ||
|
||||
filters[:media_kind] != 'all' ||
|
||||
['media_kind', 'file_byte_size'].include?(filters[:sort])
|
||||
end
|
||||
|
||||
def material_index_join_tag_name q
|
||||
q.left_joins(tag: :tag_name)
|
||||
end
|
||||
|
||||
def material_index_join_file_blob q
|
||||
q.joins(<<~SQL.squish)
|
||||
LEFT JOIN active_storage_attachments material_file_attachments
|
||||
ON material_file_attachments.record_type = 'Material'
|
||||
AND material_file_attachments.record_id = materials.id
|
||||
AND material_file_attachments.name = 'file'
|
||||
LEFT JOIN active_storage_blobs material_file_blobs
|
||||
ON material_file_blobs.id = material_file_attachments.blob_id
|
||||
SQL
|
||||
end
|
||||
|
||||
def apply_material_query q, term
|
||||
like = "%#{ ActiveRecord::Base.sanitize_sql_like(term) }%"
|
||||
q.where('tag_names.name LIKE :q OR materials.url LIKE :q OR ' \
|
||||
'material_file_blobs.filename LIKE :q',
|
||||
q: like)
|
||||
end
|
||||
|
||||
def apply_material_media_kind q, media_kind
|
||||
case media_kind
|
||||
when 'image'
|
||||
q.where('material_file_blobs.content_type LIKE ?', 'image/%')
|
||||
when 'video'
|
||||
q.where('material_file_blobs.content_type LIKE ?', 'video/%')
|
||||
when 'audio'
|
||||
q.where('material_file_blobs.content_type LIKE ?', 'audio/%')
|
||||
when 'file_other'
|
||||
q.where('material_file_attachments.id IS NOT NULL')
|
||||
.where.not('material_file_blobs.content_type LIKE ?', 'image/%')
|
||||
.where.not('material_file_blobs.content_type LIKE ?', 'video/%')
|
||||
.where.not('material_file_blobs.content_type LIKE ?', 'audio/%')
|
||||
when 'url_only'
|
||||
q.where('material_file_attachments.id IS NULL').where.not(url: [nil, ''])
|
||||
else
|
||||
q
|
||||
end
|
||||
end
|
||||
|
||||
def apply_material_tag_filter q, filters, tag_graph
|
||||
return q unless filters[:tag_id]
|
||||
|
||||
tag_ids =
|
||||
if tag_graph
|
||||
tag_graph[:scope_tag_ids]
|
||||
else
|
||||
[filters[:tag_id]]
|
||||
end
|
||||
|
||||
q.where(tag_id: tag_ids)
|
||||
end
|
||||
|
||||
def material_index_tag_graph filters
|
||||
return nil unless filters[:tag_id]
|
||||
|
||||
children_by_parent_id = Hash.new { |h, k| h[k] = [] }
|
||||
TagImplication.pluck(:parent_tag_id, :tag_id).each do |parent_id, child_id|
|
||||
children_by_parent_id[parent_id] << child_id
|
||||
end
|
||||
|
||||
scope_tag_ids =
|
||||
if filters[:include_descendants]
|
||||
collect_material_descendant_ids(filters[:tag_id], children_by_parent_id)
|
||||
else
|
||||
Set.new([filters[:tag_id]])
|
||||
end
|
||||
tags_by_id =
|
||||
Tag
|
||||
.joins(:tag_name)
|
||||
.where(id: scope_tag_ids)
|
||||
.pluck('tags.id', 'tag_names.name', 'tags.category', 'tags.deprecated_at')
|
||||
.each_with_object({ }) do |(id, name, category, deprecated_at), hash|
|
||||
hash[id] = { id:, name:, category:, deprecated: deprecated_at.present? }
|
||||
end
|
||||
|
||||
selected_tag = tags_by_id[filters[:tag_id]]
|
||||
return nil unless selected_tag
|
||||
|
||||
group_tag_id_by_tag_id = { }
|
||||
children_by_parent_id[filters[:tag_id]].each do |child_tag_id|
|
||||
assign_material_group_branch(child_tag_id,
|
||||
children_by_parent_id:,
|
||||
tags_by_id:,
|
||||
group_tag_id_by_tag_id:,
|
||||
current_group_tag_id: nil,
|
||||
seen: Set.new([filters[:tag_id]]))
|
||||
end
|
||||
|
||||
{ selected_tag_id: filters[:tag_id],
|
||||
selected_tag:,
|
||||
scope_tag_ids: scope_tag_ids.to_a,
|
||||
tags_by_id:,
|
||||
group_tag_id_by_tag_id: }
|
||||
end
|
||||
|
||||
def collect_material_descendant_ids selected_tag_id, children_by_parent_id
|
||||
ids = Set.new([selected_tag_id])
|
||||
stack = [selected_tag_id]
|
||||
|
||||
until stack.empty?
|
||||
tag_id = stack.pop
|
||||
children_by_parent_id[tag_id].each do |child_tag_id|
|
||||
next if ids.include?(child_tag_id)
|
||||
|
||||
ids << child_tag_id
|
||||
stack << child_tag_id
|
||||
end
|
||||
end
|
||||
|
||||
ids
|
||||
end
|
||||
|
||||
def assign_material_group_branch tag_id, children_by_parent_id:, tags_by_id:,
|
||||
group_tag_id_by_tag_id:, current_group_tag_id:, seen:
|
||||
return if seen.include?(tag_id)
|
||||
|
||||
tag = tags_by_id[tag_id]
|
||||
return unless tag
|
||||
|
||||
seen = seen.dup << tag_id
|
||||
next_group_tag_id = current_group_tag_id
|
||||
next_group_tag_id = tag_id if next_group_tag_id.nil? && !tag[:deprecated]
|
||||
group_tag_id_by_tag_id[tag_id] = next_group_tag_id if next_group_tag_id
|
||||
|
||||
children_by_parent_id[tag_id].each do |child_tag_id|
|
||||
assign_material_group_branch(child_tag_id,
|
||||
children_by_parent_id:,
|
||||
tags_by_id:,
|
||||
group_tag_id_by_tag_id:,
|
||||
current_group_tag_id: next_group_tag_id,
|
||||
seen:)
|
||||
end
|
||||
end
|
||||
|
||||
def material_index_groups materials, tag_graph
|
||||
return [] unless tag_graph
|
||||
|
||||
groups_by_key = { }
|
||||
|
||||
materials.each do |material|
|
||||
next unless material.tag_id
|
||||
|
||||
group_tag_id =
|
||||
if material.tag_id == tag_graph[:selected_tag_id]
|
||||
tag_graph[:selected_tag_id]
|
||||
else
|
||||
tag_graph[:group_tag_id_by_tag_id][material.tag_id] || tag_graph[:selected_tag_id]
|
||||
end
|
||||
|
||||
group_tag = tag_graph[:tags_by_id][group_tag_id]
|
||||
next unless group_tag
|
||||
|
||||
key = "tag:#{ group_tag_id }"
|
||||
group =
|
||||
groups_by_key[key] ||= { key:,
|
||||
tag: group_tag_repr(group_tag),
|
||||
material_ids: [],
|
||||
count: 0 }
|
||||
group[:material_ids] << material.id
|
||||
group[:count] += 1
|
||||
end
|
||||
|
||||
groups_by_key.values.sort_by { |group| group[:tag][:name] }
|
||||
end
|
||||
|
||||
def material_index_tag_scope filters, tag_graph
|
||||
return nil unless tag_graph
|
||||
|
||||
{
|
||||
tag: group_tag_repr(tag_graph[:selected_tag]),
|
||||
include_descendants: filters[:include_descendants]
|
||||
}
|
||||
end
|
||||
|
||||
def group_tag_repr group_tag
|
||||
{ id: group_tag[:id],
|
||||
name: group_tag[:name],
|
||||
category: group_tag[:category] }
|
||||
end
|
||||
|
||||
def material_index_order_sql filters
|
||||
direction = filters[:direction] == 'asc' ? 'ASC' : 'DESC'
|
||||
sort_sql =
|
||||
case filters[:sort]
|
||||
when 'tag_name'
|
||||
'tag_names.name'
|
||||
when 'media_kind'
|
||||
material_media_kind_sql
|
||||
when 'file_byte_size'
|
||||
'material_file_blobs.byte_size'
|
||||
when 'updated_at'
|
||||
'materials.updated_at'
|
||||
when 'version_no'
|
||||
'materials.version_no'
|
||||
when 'id'
|
||||
'materials.id'
|
||||
else
|
||||
'materials.created_at'
|
||||
end
|
||||
|
||||
"#{ sort_sql } #{ direction }, materials.id #{ direction }"
|
||||
end
|
||||
|
||||
def material_media_kind_sql
|
||||
"CASE " \
|
||||
"WHEN material_file_attachments.id IS NULL AND materials.url IS NOT NULL THEN 5 " \
|
||||
"WHEN material_file_blobs.content_type LIKE 'image/%' THEN 1 " \
|
||||
"WHEN material_file_blobs.content_type LIKE 'video/%' THEN 2 " \
|
||||
"WHEN material_file_blobs.content_type LIKE 'audio/%' THEN 3 " \
|
||||
"WHEN material_file_attachments.id IS NOT NULL THEN 4 " \
|
||||
"ELSE 6 END"
|
||||
end
|
||||
|
||||
def upsert_export_paths! material
|
||||
raw = params[:export_paths]
|
||||
return if raw.blank?
|
||||
|
||||
export_paths = raw.respond_to?(:to_unsafe_h) ? raw.to_unsafe_h : raw.to_h
|
||||
export_paths.each do |profile, export_path|
|
||||
profile = profile.to_s
|
||||
export_path = export_path.to_s.strip
|
||||
item = material.material_export_items.find_or_initialize_by(profile:)
|
||||
|
||||
if export_path.blank?
|
||||
item.destroy! if item.persisted?
|
||||
next
|
||||
end
|
||||
|
||||
item.export_path = export_path
|
||||
item.enabled = true
|
||||
item.created_by_user ||= current_user
|
||||
item.save!
|
||||
end
|
||||
end
|
||||
|
||||
def render_zip_empty
|
||||
render_unprocessable_entity('ZIP export 対象の素材がありません．')
|
||||
end
|
||||
|
||||
def render_zip_duplicate_path error
|
||||
render_unprocessable_entity("ZIP export path が重複してゐます: #{ error.message }")
|
||||
end
|
||||
|
||||
def render_zip_missing_file error
|
||||
missing_files = error.missing_files.map do |missing_file|
|
||||
{ material_id: missing_file.material_id,
|
||||
export_path: missing_file.export_path,
|
||||
blob_id: missing_file.blob_id,
|
||||
filename: missing_file.filename }
|
||||
end
|
||||
|
||||
render json: { type: 'validation_error',
|
||||
message: 'ZIP export に必要な素材ファイルが欠損しています．',
|
||||
errors: { },
|
||||
base_errors: ['ZIP export に必要な素材ファイルが欠損しています．'],
|
||||
missing_files: },
|
||||
status: :unprocessable_entity
|
||||
end
|
||||
|
||||
def build_uploaded_material_blob! file, file_sha256
|
||||
return nil unless file
|
||||
|
||||
file.tempfile.rewind
|
||||
|
||||
blob = ActiveStorage::Blob.create_and_upload!(
|
||||
io: file.tempfile,
|
||||
filename: file.original_filename,
|
||||
content_type: file.content_type,
|
||||
)
|
||||
|
||||
MaterialFileSha256.assign_metadata_sha256!(blob, file_sha256)
|
||||
|
||||
blob
|
||||
ensure
|
||||
file.tempfile.rewind if file&.tempfile
|
||||
end
|
||||
|
||||
def render_material_import_block block
|
||||
render_validation_error fields: { file: ["抑止された素材です: #{ block.reason }"] }
|
||||
end
|
||||
|
||||
def log_thumbnail_generation material, result
|
||||
material.reload
|
||||
Rails.logger.info(
|
||||
"Material thumbnail generation: material_id=#{ material.id } " \
|
||||
"result=#{ result } " \
|
||||
"thumbnail_attached=#{ material.thumbnail.attached? } " \
|
||||
"content_type=#{ material.content_type }")
|
||||
end
|
||||
end
|
||||
|
||||
@@ -82,8 +82,9 @@ class TagsController < ApplicationController
|
||||
def with_depth
|
||||
parent_tag_id = params[:parent].to_i
|
||||
parent_tag_id = nil if parent_tag_id <= 0
|
||||
material_filter = material_filter_param(default: 'any')
|
||||
|
||||
graph = build_with_depth_graph
|
||||
graph = build_with_depth_graph(material_filter)
|
||||
|
||||
tag_ids =
|
||||
if parent_tag_id
|
||||
@@ -92,19 +93,9 @@ class TagsController < ApplicationController
|
||||
visible_root_tag_ids(graph)
|
||||
end
|
||||
|
||||
tags =
|
||||
Tag
|
||||
.joins(:tag_name)
|
||||
.includes(:tag_name, :materials, tag_name: :wiki_page)
|
||||
.where(id: tag_ids)
|
||||
.order('tag_names.name')
|
||||
.distinct
|
||||
.to_a
|
||||
|
||||
render json: tags.map { |tag|
|
||||
TagRepr.base(tag).merge(has_children: visible_child_tag_ids(tag.id, graph).present?,
|
||||
children: [])
|
||||
}
|
||||
render json: tag_ids
|
||||
.sort_by { |tag_id| graph[:tags_by_id][tag_id][:name] }
|
||||
.map { |tag_id| with_depth_lightweight_repr(tag_id, graph) }
|
||||
end
|
||||
|
||||
def autocomplete
|
||||
@@ -229,13 +220,16 @@ class TagsController < ApplicationController
|
||||
def materials_by_name
|
||||
name = params[:name].to_s.strip
|
||||
return render_bad_request('name は必須です．') if name.blank?
|
||||
material_filter = material_filter_param(default: 'any')
|
||||
|
||||
tag = Tag.joins(:tag_name)
|
||||
.includes(:tag_name, :materials, tag_name: :wiki_page)
|
||||
.find_by(tag_names: { name: })
|
||||
return head :not_found unless tag
|
||||
|
||||
render json: build_tag_children(tag)
|
||||
graph = build_with_depth_graph(material_filter)
|
||||
|
||||
render json: build_tag_children(tag, graph:)
|
||||
end
|
||||
|
||||
def update_all
|
||||
@@ -348,7 +342,14 @@ class TagsController < ApplicationController
|
||||
|
||||
private
|
||||
|
||||
def build_with_depth_graph
|
||||
def material_filter_param default:
|
||||
value = params[:material_filter].to_s.presence
|
||||
return default unless ['present', 'missing', 'any'].include?(value)
|
||||
|
||||
value
|
||||
end
|
||||
|
||||
def build_with_depth_graph material_filter
|
||||
children_by_parent_id = Hash.new { |h, k| h[k] = [] }
|
||||
parent_ids_by_child_id = Hash.new { |h, k| h[k] = [] }
|
||||
|
||||
@@ -361,23 +362,37 @@ class TagsController < ApplicationController
|
||||
parent_ids_by_child_id.keys +
|
||||
Tag.where(category: ['meme', 'character', 'material']).pluck(:id)).uniq
|
||||
|
||||
tags_by_id = Tag.where(id: tag_ids)
|
||||
.pluck(:id, :category, :deprecated_at)
|
||||
.each_with_object({ }) do |(id, category, deprecated_at), h|
|
||||
h[id] = { category:, deprecated: deprecated_at.present? }
|
||||
material_tag_ids = Material.unscoped.kept.where.not(tag_id: nil).distinct.pluck(:tag_id).to_set
|
||||
|
||||
tags_by_id = Tag.joins(:tag_name)
|
||||
.where(id: tag_ids)
|
||||
.pluck('tags.id', 'tag_names.name', 'tags.category', 'tags.deprecated_at')
|
||||
.each_with_object({ }) do |(id, name, category, deprecated_at), h|
|
||||
h[id] = { name:, category:, deprecated: deprecated_at.present?,
|
||||
has_material: material_tag_ids.include?(id) }
|
||||
end
|
||||
|
||||
{ children_by_parent_id:, parent_ids_by_child_id:, tags_by_id:,
|
||||
visible_child_tag_ids_by_parent_id: { } }
|
||||
visible_child_tag_ids_by_parent_id: { },
|
||||
visible_subtree_by_tag_id: { },
|
||||
material_filter: }
|
||||
end
|
||||
|
||||
def visible_root_tag_ids graph
|
||||
graph[:tags_by_id].filter_map do |tag_id, attrs|
|
||||
next unless with_depth_visible_tag?(attrs)
|
||||
next unless visible_root_tag?(tag_id, graph)
|
||||
root_ids = Set.new
|
||||
|
||||
tag_id
|
||||
graph[:tags_by_id].each do |tag_id, attrs|
|
||||
next unless visible_root_tag?(tag_id, graph)
|
||||
next unless visible_subtree?(tag_id, graph)
|
||||
|
||||
if attrs[:deprecated]
|
||||
collect_visible_child_tag_ids(tag_id, graph, root_ids, Set.new)
|
||||
else
|
||||
root_ids << tag_id
|
||||
end
|
||||
end
|
||||
|
||||
root_ids.to_a
|
||||
end
|
||||
|
||||
def visible_root_tag? tag_id, graph
|
||||
@@ -428,25 +443,77 @@ class TagsController < ApplicationController
|
||||
return
|
||||
end
|
||||
|
||||
visible_ids << tag_id if with_depth_visible_tag?(tag)
|
||||
if with_depth_visible_tag?(tag, graph[:material_filter])
|
||||
visible_ids << tag_id
|
||||
return
|
||||
end
|
||||
|
||||
visible_ids << tag_id if graph[:children_by_parent_id][tag_id].any? { |child_tag_id|
|
||||
visible_subtree?(child_tag_id, graph)
|
||||
}
|
||||
end
|
||||
|
||||
def with_depth_visible_tag? tag
|
||||
tag[:category].in?(['meme', 'character', 'material']) && !tag[:deprecated]
|
||||
def with_depth_visible_tag? tag, material_filter
|
||||
return false unless tag[:category].in?(['meme', 'character', 'material']) && !tag[:deprecated]
|
||||
|
||||
case material_filter
|
||||
when 'present'
|
||||
tag[:has_material]
|
||||
when 'missing'
|
||||
!tag[:has_material]
|
||||
else
|
||||
true
|
||||
end
|
||||
end
|
||||
|
||||
def build_tag_children tag
|
||||
def visible_subtree? tag_id, graph
|
||||
cache = graph[:visible_subtree_by_tag_id]
|
||||
return cache[tag_id] if cache.key?(tag_id)
|
||||
|
||||
tag = graph[:tags_by_id][tag_id]
|
||||
return cache[tag_id] = false unless tag
|
||||
|
||||
if tag[:deprecated]
|
||||
return cache[tag_id] =
|
||||
graph[:children_by_parent_id][tag_id].any? { |child_tag_id|
|
||||
visible_subtree?(child_tag_id, graph)
|
||||
}
|
||||
end
|
||||
|
||||
cache[tag_id] =
|
||||
with_depth_visible_tag?(tag, graph[:material_filter]) ||
|
||||
graph[:children_by_parent_id][tag_id].any? { |child_tag_id|
|
||||
visible_subtree?(child_tag_id, graph)
|
||||
}
|
||||
end
|
||||
|
||||
def with_depth_lightweight_repr tag_id, graph
|
||||
tag = graph[:tags_by_id][tag_id]
|
||||
|
||||
{ id: tag_id,
|
||||
name: tag[:name],
|
||||
category: tag[:category],
|
||||
deprecated: tag[:deprecated],
|
||||
has_material: tag[:has_material],
|
||||
has_children: visible_child_tag_ids(tag_id, graph).present?,
|
||||
children: [] }
|
||||
end
|
||||
|
||||
def build_tag_children tag, graph: nil
|
||||
material = tag.materials.first
|
||||
file = nil
|
||||
content_type = nil
|
||||
if material&.file&.attached?
|
||||
file = rails_storage_proxy_url(material.file, only_path: false)
|
||||
content_type = material.file.blob.content_type
|
||||
tag_graph = graph && graph[:tags_by_id][tag.id]
|
||||
material =
|
||||
nil if tag_graph && !with_depth_visible_tag?(tag_graph, graph[:material_filter])
|
||||
|
||||
children = tag.children.sort_by(&:name)
|
||||
if graph
|
||||
children = children.filter { |child_tag| visible_subtree?(child_tag.id, graph) }
|
||||
end
|
||||
|
||||
TagRepr.base(tag).merge(
|
||||
children: tag.children.sort_by { _1.name }.map { build_tag_children(_1) },
|
||||
material: material.as_json&.merge(file:, content_type:))
|
||||
children: children.map { build_tag_children(_1, graph:) },
|
||||
has_material: tag_graph ? tag_graph[:has_material] : material.present?,
|
||||
material: material && MaterialRepr.base(material, host: request.base_url))
|
||||
end
|
||||
|
||||
def record_tag_version! tag, event_type:, created_by_user:, name_changed: false, wiki_page: nil
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
class Material < ApplicationRecord
|
||||
include MyDiscard
|
||||
|
||||
SOURCE_KINDS = ['uri', 'google_drive_path', 'google_drive_file',
|
||||
'legacy_drive_path'].freeze
|
||||
|
||||
default_scope -> { kept }
|
||||
|
||||
belongs_to :parent, class_name: 'Material', optional: true
|
||||
@@ -10,9 +13,18 @@ class Material < ApplicationRecord
|
||||
belongs_to :created_by_user, class_name: 'User', optional: true
|
||||
belongs_to :updated_by_user, class_name: 'User', optional: true
|
||||
|
||||
has_one_attached :file, dependent: :purge
|
||||
has_many :material_versions, dependent: :destroy
|
||||
has_many :material_export_items, dependent: :destroy
|
||||
|
||||
validates :tag_id, presence: true, uniqueness: true
|
||||
has_one_attached :file, dependent: :purge
|
||||
has_one_attached :thumbnail, dependent: :purge
|
||||
|
||||
before_validation :assign_normalized_source_key
|
||||
|
||||
validates :tag_id, uniqueness: true, allow_nil: true
|
||||
validates :source_kind,
|
||||
inclusion: { in: SOURCE_KINDS },
|
||||
allow_blank: true
|
||||
|
||||
validate :file_must_be_attached
|
||||
validate :tag_must_be_material_category
|
||||
@@ -23,8 +35,37 @@ class Material < ApplicationRecord
|
||||
file.blob.content_type
|
||||
end
|
||||
|
||||
def file_byte_size
|
||||
return nil unless file&.attached?
|
||||
|
||||
file.blob.byte_size
|
||||
end
|
||||
|
||||
def file_filename
|
||||
return nil unless file&.attached?
|
||||
|
||||
file.blob.filename.to_s
|
||||
end
|
||||
|
||||
def snapshot_export_paths
|
||||
material_export_items.order(:profile).pluck(:profile, :export_path).to_h
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def assign_normalized_source_key
|
||||
if source_kind.blank?
|
||||
self.normalized_source_key = nil
|
||||
return
|
||||
end
|
||||
|
||||
self.normalized_source_key =
|
||||
MaterialSyncSuppression.normalize_source_key(source_kind:,
|
||||
source_uri:,
|
||||
drive_path: source_path,
|
||||
drive_file_id: source_file_id)
|
||||
end
|
||||
|
||||
def file_must_be_attached
|
||||
return if url.present? || file.attached?
|
||||
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
class MaterialExportItem < ApplicationRecord
|
||||
VALID_PROFILES = ['legacy_drive'].freeze
|
||||
|
||||
belongs_to :material
|
||||
belongs_to :created_by_user, class_name: 'User', optional: true
|
||||
|
||||
validates :profile, presence: true, inclusion: { in: VALID_PROFILES }
|
||||
validates :export_path, presence: true, uniqueness: { scope: :profile }
|
||||
validates :material_id, uniqueness: { scope: :profile }
|
||||
validate :export_path_must_be_relative_safe_path
|
||||
|
||||
scope :enabled, -> { where(enabled: true) }
|
||||
|
||||
private
|
||||
|
||||
def export_path_must_be_relative_safe_path
|
||||
return if export_path.blank?
|
||||
|
||||
if export_path.start_with?('/')
|
||||
errors.add(:export_path, '絶対パスは使へません．')
|
||||
end
|
||||
|
||||
if export_path.match?(/\A[A-Za-z]:\//)
|
||||
errors.add(:export_path, '絶対パスは使へません．')
|
||||
end
|
||||
|
||||
if export_path.include?('\\')
|
||||
errors.add(:export_path, '/ 区切りで指定してください．')
|
||||
end
|
||||
|
||||
if export_path.include?("\0")
|
||||
errors.add(:export_path, 'NUL は使へません．')
|
||||
end
|
||||
|
||||
parts = export_path.split('/')
|
||||
if export_path.include?('//')
|
||||
errors.add(:export_path, '空の path segment は使へません．')
|
||||
end
|
||||
|
||||
if parts.any? { |part| part.in?(['.', '..']) }
|
||||
errors.add(:export_path, '.. は使へません．')
|
||||
end
|
||||
|
||||
if export_path.end_with?('/')
|
||||
errors.add(:export_path, 'directory path は使へません．')
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,29 @@
|
||||
class MaterialImportBlock < ApplicationRecord
|
||||
MATCH_KINDS = ['sha256', 'exact_path', 'path_prefix', 'manual'].freeze
|
||||
REASONS = [
|
||||
'copyright_high_risk',
|
||||
'copyright_takedown',
|
||||
'adult_or_sensitive',
|
||||
'personal_information',
|
||||
'malware_or_dangerous_file',
|
||||
'duplicate_or_low_quality',
|
||||
'source_owner_request',
|
||||
'other'
|
||||
].freeze
|
||||
|
||||
belongs_to :created_by_user, class_name: 'User', optional: true
|
||||
|
||||
validates :match_kind, presence: true, inclusion: { in: MATCH_KINDS }
|
||||
validates :reason, presence: true, inclusion: { in: REASONS }
|
||||
validates :sha256, length: { is: 64 }, allow_blank: true
|
||||
validate :match_value_must_be_present
|
||||
|
||||
private
|
||||
|
||||
def match_value_must_be_present
|
||||
return if match_kind == 'manual'
|
||||
return if sha256.present? || external_path_pattern.present?
|
||||
|
||||
errors.add(:base, 'sha256 または external_path_pattern は必須です．')
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,45 @@
|
||||
class MaterialSyncSource < ApplicationRecord
|
||||
SOURCE_KINDS = ['uri', 'google_drive_path', 'google_drive_file', 'legacy_drive_path'].freeze
|
||||
|
||||
belongs_to :created_by_user, class_name: 'User', optional: true
|
||||
belongs_to :updated_by_user, class_name: 'User', optional: true
|
||||
|
||||
validates :name, presence: true
|
||||
validates :source_kind,
|
||||
presence: true,
|
||||
inclusion: { in: SOURCE_KINDS }
|
||||
validates :profile, presence: true, inclusion: { in: MaterialExportItem::VALID_PROFILES }
|
||||
validate :source_value_must_be_present
|
||||
|
||||
scope :enabled, -> { where(enabled: true) }
|
||||
|
||||
def normalized_source_key
|
||||
MaterialSyncSuppression.normalize_source_key(source_kind:,
|
||||
source_uri:,
|
||||
source_path:,
|
||||
source_file_id:)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def source_value_must_be_present
|
||||
return if source_value_present?
|
||||
|
||||
errors.add(:base, '同期元を指定してください．')
|
||||
end
|
||||
|
||||
def source_value_present?
|
||||
case source_kind
|
||||
when 'uri'
|
||||
source_uri.present?
|
||||
when 'google_drive_file'
|
||||
source_file_id.present? || source_uri.present?
|
||||
when 'google_drive_path'
|
||||
source_file_id.present? || source_uri.present?
|
||||
when 'legacy_drive_path'
|
||||
source_path.present?
|
||||
else
|
||||
normalized_source_key.present?
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,105 @@
|
||||
class MaterialSyncSuppression < ApplicationRecord
|
||||
SOURCE_KINDS = [
|
||||
'uri',
|
||||
'google_drive_path',
|
||||
'google_drive_path_prefix',
|
||||
'google_drive_file',
|
||||
'legacy_drive_path',
|
||||
'legacy_drive_path_prefix'
|
||||
].freeze
|
||||
|
||||
REASONS = [
|
||||
'copyright_high_risk',
|
||||
'copyright_takedown',
|
||||
'adult_or_sensitive',
|
||||
'personal_information',
|
||||
'malware_or_dangerous_file',
|
||||
'duplicate_or_low_quality',
|
||||
'source_owner_request',
|
||||
'other'
|
||||
].freeze
|
||||
|
||||
belongs_to :created_by_user, class_name: 'User', optional: true
|
||||
|
||||
before_validation :assign_normalized_source_key
|
||||
|
||||
validates :source_kind, presence: true, inclusion: { in: SOURCE_KINDS }
|
||||
validates :reason, presence: true, inclusion: { in: REASONS }
|
||||
validates :normalized_source_key, presence: true, uniqueness: true
|
||||
validate :source_value_must_be_present
|
||||
validate :drive_path_must_be_safe_relative_path
|
||||
|
||||
def self.normalize_source_key source_kind:,
|
||||
source_uri: nil,
|
||||
drive_path: nil,
|
||||
drive_file_id: nil,
|
||||
source_path: nil,
|
||||
source_file_id: nil
|
||||
kind = source_kind.to_s
|
||||
value =
|
||||
case kind
|
||||
when 'uri'
|
||||
source_uri.to_s.strip
|
||||
when 'google_drive_path', 'google_drive_path_prefix',
|
||||
'legacy_drive_path', 'legacy_drive_path_prefix'
|
||||
(source_path || drive_path).to_s.strip.gsub(%r{/+}, '/')
|
||||
when 'google_drive_file'
|
||||
(source_file_id || drive_file_id).to_s.strip
|
||||
else
|
||||
''
|
||||
end
|
||||
|
||||
return nil if kind.blank? || value.blank?
|
||||
|
||||
"#{ kind }:#{ value }"
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def assign_normalized_source_key
|
||||
normalize_path_fields!
|
||||
self.normalized_source_key =
|
||||
self.class.normalize_source_key(source_kind:,
|
||||
source_uri:,
|
||||
drive_path:,
|
||||
drive_file_id:)
|
||||
end
|
||||
|
||||
def source_value_must_be_present
|
||||
return if normalized_source_key.present?
|
||||
|
||||
errors.add(:base, '同期元を指定してください．')
|
||||
end
|
||||
|
||||
def drive_path_must_be_safe_relative_path
|
||||
return unless source_kind.in?(
|
||||
['google_drive_path', 'google_drive_path_prefix',
|
||||
'legacy_drive_path', 'legacy_drive_path_prefix'])
|
||||
|
||||
value = drive_path.to_s
|
||||
return if value.blank?
|
||||
|
||||
if value.start_with?('/') || value.match?(/\A[A-Za-z]:\//)
|
||||
errors.add(:drive_path, '相対 path を指定してください．')
|
||||
end
|
||||
if value.start_with?('My Drive/', 'マイドライブ/')
|
||||
errors.add(:drive_path, '同期元フォルダからの相対 path を指定してください．')
|
||||
end
|
||||
errors.add(:drive_path, 'NUL は使へません．') if value.include?("\0")
|
||||
errors.add(:drive_path, '連続スラッシュは使へません．') if value.include?('//')
|
||||
errors.add(:drive_path, '末尾スラッシュは使へません．') if value.end_with?('/')
|
||||
|
||||
parts = value.split('/')
|
||||
if parts.any? { |part| part.in?(['.', '..']) }
|
||||
errors.add(:drive_path, '. や .. は使へません．')
|
||||
end
|
||||
end
|
||||
|
||||
def normalize_path_fields!
|
||||
return unless source_kind.in?(
|
||||
['google_drive_path', 'google_drive_path_prefix',
|
||||
'legacy_drive_path', 'legacy_drive_path_prefix'])
|
||||
|
||||
self.drive_path = drive_path.to_s.tr('\\', '/').strip.presence
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,17 @@
|
||||
class MaterialVersion < ApplicationRecord
|
||||
EVENT_TYPE_MAP = { create: 'create',
|
||||
update: 'update',
|
||||
discard: 'discard',
|
||||
restore: 'restore' }.freeze
|
||||
|
||||
include VersionRecord
|
||||
|
||||
belongs_to :material
|
||||
belongs_to :tag, optional: true
|
||||
belongs_to :parent, class_name: 'Material', optional: true
|
||||
belongs_to :updated_by_user, class_name: 'User', optional: true
|
||||
|
||||
def export_paths_hash
|
||||
export_paths_json || {}
|
||||
end
|
||||
end
|
||||
@@ -1,15 +1,23 @@
|
||||
module VersionRecord
|
||||
extend ActiveSupport::Concern
|
||||
|
||||
DEFAULT_EVENT_TYPE_MAP = { create: 'create',
|
||||
update: 'update',
|
||||
discard: 'discard',
|
||||
restore: 'restore' }.freeze
|
||||
|
||||
def readonly? = persisted?
|
||||
|
||||
included do
|
||||
event_type_map = if const_defined?(:EVENT_TYPE_MAP, false)
|
||||
const_get(:EVENT_TYPE_MAP)
|
||||
else
|
||||
DEFAULT_EVENT_TYPE_MAP
|
||||
end
|
||||
|
||||
belongs_to :created_by_user, class_name: 'User', optional: true
|
||||
|
||||
enum :event_type, { create: 'create',
|
||||
update: 'update',
|
||||
discard: 'discard',
|
||||
restore: 'restore' }, prefix: true, validate: true
|
||||
enum :event_type, event_type_map, prefix: true, validate: true
|
||||
|
||||
validates :version_no, presence: true, numericality: { only_integer: true, greater_than: 0 }
|
||||
validates :event_type, presence: true
|
||||
|
||||
@@ -2,7 +2,9 @@
|
||||
|
||||
|
||||
module MaterialRepr
|
||||
BASE = { only: [:id, :url, :created_at, :updated_at],
|
||||
BASE = { only: [:id, :url, :version_no, :source_kind, :source_uri,
|
||||
:source_path, :source_file_id, :normalized_source_key,
|
||||
:created_at, :updated_at],
|
||||
methods: [:content_type],
|
||||
include: { tag: TagRepr::BASE,
|
||||
created_by_user: UserRepr::BASE,
|
||||
@@ -15,10 +17,93 @@ module MaterialRepr
|
||||
file: if material.file.attached?
|
||||
Rails.application.routes.url_helpers.rails_storage_proxy_url(
|
||||
material.file, host:)
|
||||
end)
|
||||
end,
|
||||
thumbnail: thumbnail_url(material, host:),
|
||||
thumbnail_fallback_text: thumbnail_fallback_text(material),
|
||||
thumbnail_fallback_kind: thumbnail_fallback_kind(material),
|
||||
media_kind: media_kind(material),
|
||||
file_byte_size: material.file_byte_size,
|
||||
export_paths: export_paths(material),
|
||||
export_items: export_items(material))
|
||||
end
|
||||
|
||||
def many materials, host:
|
||||
materials.map { |m| base(m, host:) }
|
||||
end
|
||||
|
||||
def list material, host:
|
||||
{ id: material.id,
|
||||
version_no: material.version_no,
|
||||
url: material.url,
|
||||
source_kind: material.source_kind,
|
||||
source_uri: material.source_uri,
|
||||
source_path: material.source_path,
|
||||
source_file_id: material.source_file_id,
|
||||
normalized_source_key: material.normalized_source_key,
|
||||
tag: compact_tag(material.tag),
|
||||
thumbnail: thumbnail_url(material, host:),
|
||||
thumbnail_fallback_text: thumbnail_fallback_text(material),
|
||||
thumbnail_fallback_kind: thumbnail_fallback_kind(material),
|
||||
media_kind: media_kind(material),
|
||||
content_type: material.content_type,
|
||||
file_byte_size: material.file_byte_size,
|
||||
created_at: material.created_at,
|
||||
updated_at: material.updated_at,
|
||||
export_paths: export_paths(material),
|
||||
export_items: export_items(material) }
|
||||
end
|
||||
|
||||
def list_many materials, host:
|
||||
materials.map { |m| list(m, host:) }
|
||||
end
|
||||
|
||||
def export_paths material
|
||||
material.material_export_items.each_with_object({ }) do |item, hash|
|
||||
hash[item.profile] = item.enabled ? item.export_path : ''
|
||||
end
|
||||
end
|
||||
|
||||
def export_items material
|
||||
material.material_export_items.map do |item|
|
||||
{ id: item.id,
|
||||
profile: item.profile,
|
||||
export_path: item.export_path,
|
||||
enabled: item.enabled }
|
||||
end
|
||||
end
|
||||
|
||||
def thumbnail_url material, host:
|
||||
return nil unless material.thumbnail.attached?
|
||||
|
||||
Rails.application.routes.url_helpers.rails_storage_proxy_url(
|
||||
material.thumbnail, host:)
|
||||
end
|
||||
|
||||
def thumbnail_fallback_text material
|
||||
material.tag&.name || material.created_at&.strftime('%Y-%m-%d')
|
||||
end
|
||||
|
||||
def thumbnail_fallback_kind material
|
||||
material.tag.present? ? 'tag_name' : 'created_at'
|
||||
end
|
||||
|
||||
def media_kind material
|
||||
return 'url_only' unless material.file.attached?
|
||||
|
||||
content_type = material.file.blob.content_type.to_s
|
||||
return 'image' if content_type.start_with?('image/')
|
||||
return 'video' if content_type.start_with?('video/')
|
||||
return 'audio' if content_type.start_with?('audio/')
|
||||
|
||||
'file_other'
|
||||
end
|
||||
|
||||
def compact_tag tag
|
||||
return nil unless tag
|
||||
|
||||
{ id: tag.id,
|
||||
name: tag.name,
|
||||
category: tag.category,
|
||||
deprecated_at: tag.deprecated_at }
|
||||
end
|
||||
end
|
||||
|
||||
@@ -0,0 +1,233 @@
|
||||
require 'json'
|
||||
require 'jwt'
|
||||
require 'net/http'
|
||||
require 'openssl'
|
||||
require 'tempfile'
|
||||
require 'uri'
|
||||
|
||||
|
||||
module GoogleDrive
|
||||
class ApiClient
|
||||
DRIVE_ENDPOINT = 'https://www.googleapis.com/drive/v3'
|
||||
TOKEN_ENDPOINT = 'https://oauth2.googleapis.com/token'
|
||||
FOLDER_MIME_TYPE = 'application/vnd.google-apps.folder'
|
||||
NATIVE_FILE_MIME_TYPE_PREFIX = 'application/vnd.google-apps.'
|
||||
DRIVE_SCOPE = 'https://www.googleapis.com/auth/drive.readonly'
|
||||
|
||||
def initialize service_account_email: ENV['GOOGLE_DRIVE_SERVICE_ACCOUNT_EMAIL'],
|
||||
private_key: ENV['GOOGLE_DRIVE_PRIVATE_KEY'],
|
||||
private_key_path: ENV['GOOGLE_DRIVE_PRIVATE_KEY_PATH'],
|
||||
subject: ENV['GOOGLE_DRIVE_SUBJECT']
|
||||
@service_account_email = service_account_email.to_s
|
||||
@private_key = load_private_key(private_key, private_key_path)
|
||||
@subject = subject.to_s.presence
|
||||
@access_token = nil
|
||||
end
|
||||
|
||||
def list_material_files_under_folder folder_id
|
||||
files = []
|
||||
each_material_file_under_folder(folder_id) { |entry| files << entry }
|
||||
files
|
||||
end
|
||||
|
||||
def each_material_file_under_folder folder_id
|
||||
return enum_for(__method__, folder_id) unless block_given?
|
||||
|
||||
walk_folder(folder_id, nil) do |entry, relative_path|
|
||||
next if entry['mimeType'] == FOLDER_MIME_TYPE
|
||||
next if native_file?(entry['mimeType'])
|
||||
|
||||
yield build_file_entry(entry, relative_path)
|
||||
end
|
||||
end
|
||||
|
||||
def fetch_material_file file_id
|
||||
metadata = get_file(file_id)
|
||||
return nil if native_file?(metadata['mimeType'])
|
||||
|
||||
build_file_entry(metadata, metadata['name'])
|
||||
end
|
||||
|
||||
def download_to_tempfile file_id, filename:
|
||||
tempfile = Tempfile.new(['material-sync', File.extname(filename.to_s)])
|
||||
tempfile.binmode
|
||||
request_binary("/files/#{ file_id }",
|
||||
{ alt: 'media', supportsAllDrives: true }) do |chunk|
|
||||
tempfile.write(chunk.b)
|
||||
end
|
||||
tempfile.rewind
|
||||
tempfile
|
||||
rescue StandardError
|
||||
tempfile&.close!
|
||||
raise
|
||||
end
|
||||
|
||||
def extract_file_id value
|
||||
raw = value.to_s.strip
|
||||
return nil if raw.blank?
|
||||
return raw unless raw.include?('/')
|
||||
|
||||
uri = URI.parse(raw)
|
||||
return uri.query.to_s[%r{(?:^|&)id=([^&]+)}, 1] if uri.query.present?
|
||||
return uri.path[%r{/folders/([^/]+)}, 1] if uri.path.include?('/folders/')
|
||||
|
||||
uri.path[%r{/d/([^/]+)}, 1]
|
||||
rescue URI::InvalidURIError
|
||||
nil
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def walk_folder folder_id, prefix, &block
|
||||
list_children(folder_id).each do |entry|
|
||||
relative_path = MaterialSyncExportPath.build(prefix:,
|
||||
relative_path: entry['name'])
|
||||
if entry['mimeType'] == FOLDER_MIME_TYPE
|
||||
walk_folder(entry['id'], relative_path, &block)
|
||||
next
|
||||
end
|
||||
|
||||
block.call(entry, relative_path)
|
||||
end
|
||||
end
|
||||
|
||||
def build_file_entry entry, relative_path
|
||||
{ id: entry['id'],
|
||||
name: entry['name'],
|
||||
mime_type: entry['mimeType'],
|
||||
relative_path: MaterialSyncExportPath.normalize_path(relative_path),
|
||||
sha256_checksum: entry['sha256Checksum'],
|
||||
web_view_link: entry['webViewLink'],
|
||||
web_content_link: entry['webContentLink'] }
|
||||
end
|
||||
|
||||
def list_children folder_id
|
||||
files = []
|
||||
page_token = nil
|
||||
|
||||
loop do
|
||||
response =
|
||||
request_json('/files', {
|
||||
q: "'#{ folder_id }' in parents and trashed = false",
|
||||
fields: 'nextPageToken,files(id,name,mimeType,sha256Checksum,' \
|
||||
'webViewLink,webContentLink)',
|
||||
orderBy: 'folder,name',
|
||||
pageSize: 1000,
|
||||
supportsAllDrives: true,
|
||||
includeItemsFromAllDrives: true,
|
||||
pageToken: page_token }.compact)
|
||||
files.concat(response.fetch('files'))
|
||||
page_token = response['nextPageToken']
|
||||
break if page_token.blank?
|
||||
end
|
||||
|
||||
files
|
||||
end
|
||||
|
||||
def get_file file_id
|
||||
request_json("/files/#{ file_id }",
|
||||
fields: 'id,name,mimeType,sha256Checksum,webViewLink,webContentLink',
|
||||
supportsAllDrives: true)
|
||||
end
|
||||
|
||||
def request_json path, params = {}
|
||||
response = request(:get, path, params:)
|
||||
unless response.is_a?(Net::HTTPSuccess)
|
||||
raise "Google Drive API error: #{ response.code } #{ response.body }"
|
||||
end
|
||||
|
||||
JSON.parse(response.body)
|
||||
end
|
||||
|
||||
def request_binary path, params = {}
|
||||
request(:get, path, params:) do |response|
|
||||
unless response.is_a?(Net::HTTPSuccess)
|
||||
raise "Google Drive download error: #{ response.code } #{ response.body }"
|
||||
end
|
||||
|
||||
response.read_body do |chunk|
|
||||
yield chunk.b
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def request method, path, params: nil
|
||||
uri = URI(DRIVE_ENDPOINT + path)
|
||||
uri.query = URI.encode_www_form(params) if params.present?
|
||||
|
||||
klass =
|
||||
case method
|
||||
when :get
|
||||
Net::HTTP::Get
|
||||
else
|
||||
raise ArgumentError, "Unsupported Google Drive request method: #{ method }"
|
||||
end
|
||||
request = klass.new(uri)
|
||||
request['Authorization'] = "Bearer #{ access_token }"
|
||||
|
||||
Net::HTTP.start(uri.host,
|
||||
uri.port,
|
||||
use_ssl: true,
|
||||
open_timeout: 10,
|
||||
read_timeout: 60) do |http|
|
||||
if block_given?
|
||||
http.request(request) do |response|
|
||||
return yield response
|
||||
end
|
||||
else
|
||||
http.request(request)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def access_token
|
||||
@access_token ||= fetch_access_token
|
||||
end
|
||||
|
||||
def fetch_access_token
|
||||
raise 'GOOGLE_DRIVE_SERVICE_ACCOUNT_EMAIL is required' if @service_account_email.blank?
|
||||
raise 'Google Drive private key is required' if @private_key.blank?
|
||||
|
||||
payload = { iss: @service_account_email,
|
||||
scope: DRIVE_SCOPE,
|
||||
aud: TOKEN_ENDPOINT,
|
||||
exp: 1.hour.from_now.to_i,
|
||||
iat: Time.current.to_i }
|
||||
payload[:sub] = @subject if @subject.present?
|
||||
|
||||
assertion = JWT.encode(payload, @private_key, 'RS256')
|
||||
uri = URI(TOKEN_ENDPOINT)
|
||||
request = Net::HTTP::Post.new(uri)
|
||||
request.set_form_data(
|
||||
grant_type: 'urn:ietf:params:oauth:grant-type:jwt-bearer',
|
||||
assertion:)
|
||||
|
||||
response =
|
||||
Net::HTTP.start(uri.host,
|
||||
uri.port,
|
||||
use_ssl: true,
|
||||
open_timeout: 10,
|
||||
read_timeout: 30) do |http|
|
||||
http.request(request)
|
||||
end
|
||||
unless response.is_a?(Net::HTTPSuccess)
|
||||
raise "Google OAuth error: #{ response.code } #{ response.body }"
|
||||
end
|
||||
|
||||
JSON.parse(response.body).fetch('access_token')
|
||||
end
|
||||
|
||||
def load_private_key private_key, private_key_path
|
||||
raw = private_key.to_s
|
||||
raw = raw.gsub('\n', "\n") if raw.present?
|
||||
raw = File.read(private_key_path) if raw.blank? && private_key_path.present?
|
||||
return nil if raw.blank?
|
||||
|
||||
OpenSSL::PKey::RSA.new(raw)
|
||||
end
|
||||
|
||||
def native_file? mime_type
|
||||
mime_type.to_s.start_with?(NATIVE_FILE_MIME_TYPE_PREFIX)
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,57 @@
|
||||
require 'digest'
|
||||
require 'json'
|
||||
|
||||
class MaterialFileSha256
|
||||
def self.blob_metadata blob
|
||||
metadata = blob.metadata
|
||||
return metadata if metadata.is_a?(Hash)
|
||||
return { } if metadata.blank?
|
||||
|
||||
JSON.parse(metadata)
|
||||
rescue JSON::ParserError
|
||||
{ }
|
||||
end
|
||||
|
||||
def self.metadata_sha256 blob
|
||||
blob_metadata(blob)['sha256'].presence
|
||||
end
|
||||
|
||||
def self.assign_metadata_sha256! blob, sha256
|
||||
return if sha256.blank?
|
||||
|
||||
metadata = blob_metadata(blob)
|
||||
metadata['sha256'] = sha256
|
||||
blob.metadata = metadata
|
||||
blob.save! if blob.changed?
|
||||
end
|
||||
|
||||
def self.from_blob blob, allow_download: false
|
||||
sha256 = metadata_sha256(blob)
|
||||
return sha256 if sha256.present?
|
||||
return nil unless allow_download
|
||||
|
||||
begin
|
||||
blob.open do |file|
|
||||
sha256 = Digest::SHA256.file(file.path).hexdigest
|
||||
assign_metadata_sha256!(blob, sha256)
|
||||
sha256
|
||||
end
|
||||
rescue ActiveStorage::FileNotFoundError, ArgumentError => error
|
||||
Rails.logger.warn(
|
||||
"MaterialFileSha256.from_blob failed for blob_id=#{blob.id}: " \
|
||||
"#{error.class}: #{error.message}",
|
||||
)
|
||||
nil
|
||||
end
|
||||
end
|
||||
|
||||
def self.from_upload upload
|
||||
tempfile = upload&.tempfile
|
||||
return nil unless tempfile
|
||||
|
||||
tempfile.rewind
|
||||
Digest::SHA256.file(tempfile.path).hexdigest.tap do
|
||||
tempfile.rewind
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,7 @@
|
||||
class MaterialImportBlockMatcher
|
||||
def self.match_for_sha256 sha256
|
||||
return nil if sha256.blank?
|
||||
|
||||
MaterialImportBlock.find_by(match_kind: 'sha256', sha256:)
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,56 @@
|
||||
class MaterialSyncExportPath
|
||||
class << self
|
||||
def build prefix: nil, relative_path: nil, filename: nil, source_file_id: nil
|
||||
path = normalize_path(relative_path)
|
||||
path = fallback_filename(filename, source_file_id) if path.blank?
|
||||
|
||||
normalized_prefix = normalize_path(prefix)
|
||||
return path if normalized_prefix.blank?
|
||||
return normalized_prefix if path.blank?
|
||||
|
||||
[normalized_prefix, path].join('/')
|
||||
end
|
||||
|
||||
def normalize_path path
|
||||
raw = path.to_s.delete("\0").tr('\\', '/').strip
|
||||
raw = raw.sub(%r{\A[A-Za-z]:/+}, '')
|
||||
raw = raw.sub(%r{\A/+}, '')
|
||||
return nil if raw.blank?
|
||||
|
||||
segments = raw.split('/').filter_map do |segment|
|
||||
normalized = normalize_segment(segment)
|
||||
normalized.presence
|
||||
end
|
||||
return nil if segments.empty?
|
||||
|
||||
segments.join('/')
|
||||
end
|
||||
|
||||
def uniquify path, source_file_id
|
||||
normalized = normalize_path(path)
|
||||
return normalized if normalized.blank? || source_file_id.blank?
|
||||
|
||||
ext = File.extname(normalized)
|
||||
base = ext.present? ? normalized.delete_suffix(ext) : normalized
|
||||
"#{ base }--#{ source_file_id }#{ ext }"
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def fallback_filename filename, source_file_id
|
||||
name = normalize_path(filename)
|
||||
return name if name.present?
|
||||
return nil if source_file_id.blank?
|
||||
|
||||
source_file_id.to_s
|
||||
end
|
||||
|
||||
def normalize_segment segment
|
||||
cleaned = segment.to_s.delete("\0").tr('\\/', '_').strip
|
||||
return nil if cleaned.blank? || cleaned == '.'
|
||||
return '__' if cleaned == '..'
|
||||
|
||||
cleaned
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,259 @@
|
||||
require 'digest'
|
||||
|
||||
class MaterialSyncImporter
|
||||
Result = Struct.new(:material, :action, :suppressed, :suppression, keyword_init: true)
|
||||
|
||||
def self.import! attributes
|
||||
new(attributes).import!
|
||||
end
|
||||
|
||||
def initialize attributes
|
||||
@attributes = attributes
|
||||
end
|
||||
|
||||
def import!
|
||||
source = source_attributes
|
||||
key = normalized_source_key(source)
|
||||
raise ArgumentError, 'normalized_source_key is required for material sync' if key.blank?
|
||||
|
||||
suppression = suppression_for(source)
|
||||
if suppression
|
||||
return Result.new(material: nil,
|
||||
action: :suppressed,
|
||||
suppressed: true,
|
||||
suppression:)
|
||||
end
|
||||
|
||||
material =
|
||||
Material
|
||||
.unscoped
|
||||
.find_or_initialize_by(normalized_source_key: key)
|
||||
if unchanged?(material, source)
|
||||
return Result.new(material:,
|
||||
action: :unchanged,
|
||||
suppressed: false,
|
||||
suppression: nil)
|
||||
end
|
||||
|
||||
event_type =
|
||||
if material.new_record?
|
||||
:create
|
||||
elsif material.discarded?
|
||||
:restore
|
||||
else
|
||||
:update
|
||||
end
|
||||
uploaded_blob = uploaded_blob!
|
||||
if @import_block
|
||||
return Result.new(material: nil,
|
||||
action: :suppressed,
|
||||
suppressed: true,
|
||||
suppression: @import_block)
|
||||
end
|
||||
|
||||
Material.transaction do
|
||||
if material.persisted?
|
||||
MaterialVersionRecorder.ensure_snapshot!(material,
|
||||
created_by_user: @attributes[:updated_by_user])
|
||||
end
|
||||
material.assign_attributes(material_attributes_for(material).merge(source))
|
||||
material.discarded_at = nil if material.respond_to?(:discarded_at=)
|
||||
material.file.attach(uploaded_blob) if uploaded_blob
|
||||
material.save!
|
||||
upsert_export_item!(material)
|
||||
MaterialVersionRecorder.record!(material:,
|
||||
event_type:,
|
||||
created_by_user: @attributes[:updated_by_user])
|
||||
end
|
||||
MaterialThumbnailGenerator.generate!(material)
|
||||
|
||||
Result.new(material:,
|
||||
action: event_type == :create ? :imported : :updated,
|
||||
suppressed: false,
|
||||
suppression: nil)
|
||||
rescue StandardError
|
||||
uploaded_blob&.purge_later
|
||||
raise
|
||||
ensure
|
||||
close_tempfile!
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def source_attributes
|
||||
{ source_kind: @attributes[:source_kind],
|
||||
source_uri: @attributes[:source_uri],
|
||||
source_path: @attributes[:source_path],
|
||||
source_file_id: @attributes[:source_file_id] }
|
||||
end
|
||||
|
||||
def normalized_source_key source
|
||||
MaterialSyncSuppression.normalize_source_key(source_kind: source[:source_kind],
|
||||
source_uri: source[:source_uri],
|
||||
source_path: source[:source_path],
|
||||
source_file_id: source[:source_file_id])
|
||||
end
|
||||
|
||||
def material_attributes_for material
|
||||
attrs =
|
||||
@attributes
|
||||
.except(:source_kind, :source_uri, :source_path, :source_file_id,
|
||||
:file_blob, :file_tempfile, :file_downloader,
|
||||
:filename, :content_type,
|
||||
:file_sha256, :export_path, :profile, :tag, :url)
|
||||
|
||||
if material.new_record?
|
||||
attrs[:tag] = @attributes[:tag] if @attributes.key?(:tag)
|
||||
attrs[:url] = @attributes[:url] if @attributes.key?(:url)
|
||||
return attrs
|
||||
end
|
||||
|
||||
if @attributes.key?(:tag) && @attributes[:tag].present?
|
||||
attrs[:tag] = @attributes[:tag]
|
||||
end
|
||||
if @attributes.key?(:url) && @attributes[:url].present?
|
||||
attrs[:url] = @attributes[:url]
|
||||
end
|
||||
|
||||
attrs
|
||||
end
|
||||
|
||||
def suppression_for source
|
||||
if source[:source_kind] == 'google_drive_file' && source[:source_path].present?
|
||||
return MaterialSyncSuppressionMatcher.match_google_drive_candidate(
|
||||
drive_file_id: source[:source_file_id],
|
||||
relative_path: source[:source_path])
|
||||
end
|
||||
|
||||
MaterialSyncSuppressionMatcher.match(**source)
|
||||
end
|
||||
|
||||
def uploaded_blob!
|
||||
return @attributes[:file_blob] if @attributes[:file_blob]
|
||||
|
||||
tempfile = upload_tempfile
|
||||
return nil unless tempfile
|
||||
|
||||
tempfile.rewind
|
||||
file_sha256 = @attributes[:file_sha256] || Digest::SHA256.file(tempfile.path).hexdigest
|
||||
@attributes[:file_sha256] = file_sha256
|
||||
|
||||
@import_block = MaterialImportBlockMatcher.match_for_sha256(file_sha256)
|
||||
if @import_block
|
||||
tempfile.rewind
|
||||
return nil
|
||||
end
|
||||
|
||||
blob = ActiveStorage::Blob.create_and_upload!(
|
||||
io: tempfile,
|
||||
filename: @attributes[:filename],
|
||||
content_type: @attributes[:content_type])
|
||||
MaterialFileSha256.assign_metadata_sha256!(blob, file_sha256)
|
||||
tempfile.rewind
|
||||
blob
|
||||
end
|
||||
|
||||
def upload_tempfile
|
||||
return @upload_tempfile if defined?(@upload_tempfile)
|
||||
|
||||
@upload_tempfile = @attributes[:file_tempfile]
|
||||
if @upload_tempfile.blank? && @attributes[:file_downloader]
|
||||
@upload_tempfile = @attributes[:file_downloader].call
|
||||
end
|
||||
@upload_tempfile
|
||||
end
|
||||
|
||||
def upsert_export_item! material
|
||||
export_path = resolved_export_path(material)
|
||||
return if export_path.blank?
|
||||
|
||||
item = material.material_export_items.find_or_initialize_by(profile: effective_profile)
|
||||
item.export_path = export_path
|
||||
item.enabled = true
|
||||
item.created_by_user ||= @attributes[:created_by_user]
|
||||
item.save!
|
||||
end
|
||||
|
||||
def export_path_taken_by_other? material, profile, export_path
|
||||
MaterialExportItem
|
||||
.where(profile:, export_path:)
|
||||
.where.not(material_id: material.id)
|
||||
.exists?
|
||||
end
|
||||
|
||||
def close_tempfile!
|
||||
tempfile = upload_tempfile
|
||||
return unless tempfile
|
||||
|
||||
tempfile.close! unless tempfile.closed?
|
||||
rescue StandardError
|
||||
nil
|
||||
end
|
||||
|
||||
def unchanged? material, source
|
||||
return false if material.new_record? || material.discarded?
|
||||
return false unless same_source_attributes?(material, source)
|
||||
return false unless same_export_path?(material)
|
||||
return false unless same_material_attributes?(material)
|
||||
|
||||
same_file_snapshot?(material)
|
||||
end
|
||||
|
||||
def same_source_attributes? material, source
|
||||
material.source_kind == source[:source_kind] \
|
||||
&& material.source_uri == source[:source_uri] \
|
||||
&& material.source_path == source[:source_path] \
|
||||
&& material.source_file_id == source[:source_file_id]
|
||||
end
|
||||
|
||||
def same_export_path? material
|
||||
expected = resolved_export_path(material)
|
||||
current = material.material_export_items.find { |item| item.profile == effective_profile }
|
||||
|
||||
return current.blank? if expected.blank?
|
||||
|
||||
current&.enabled && current.export_path == expected
|
||||
end
|
||||
|
||||
def same_material_attributes? material
|
||||
same_tag_attribute?(material) && same_url_attribute?(material)
|
||||
end
|
||||
|
||||
def same_file_snapshot? material
|
||||
expected_sha256 = @attributes[:file_sha256].to_s.presence
|
||||
|
||||
if expected_sha256.present?
|
||||
return false unless material.file.attached?
|
||||
|
||||
return MaterialFileSha256.metadata_sha256(material.file.blob) == expected_sha256
|
||||
end
|
||||
|
||||
!material.file.attached?
|
||||
end
|
||||
|
||||
def effective_profile
|
||||
@attributes[:profile].presence || 'legacy_drive'
|
||||
end
|
||||
|
||||
def resolved_export_path material
|
||||
export_path = @attributes[:export_path].to_s.strip
|
||||
return export_path if export_path.blank?
|
||||
return export_path unless export_path_taken_by_other?(material, effective_profile, export_path)
|
||||
|
||||
MaterialSyncExportPath.uniquify(export_path, @attributes[:source_file_id])
|
||||
end
|
||||
|
||||
def same_tag_attribute? material
|
||||
return true unless @attributes.key?(:tag)
|
||||
return true if @attributes[:tag].blank?
|
||||
|
||||
material.tag_id == @attributes[:tag].id
|
||||
end
|
||||
|
||||
def same_url_attribute? material
|
||||
return true unless @attributes.key?(:url)
|
||||
return true if @attributes[:url].blank?
|
||||
|
||||
material.url == @attributes[:url]
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,251 @@
|
||||
class MaterialSyncRunner
|
||||
Result = Struct.new(:imported, :updated, :unchanged, :suppressed, :failed,
|
||||
:errors, keyword_init: true)
|
||||
|
||||
def self.sync_enabled!
|
||||
results = MaterialSyncSource.enabled.order(:id).map { |source| new(source).sync! }
|
||||
|
||||
Result.new(imported: results.sum(&:imported),
|
||||
updated: results.sum(&:updated),
|
||||
unchanged: results.sum(&:unchanged),
|
||||
suppressed: results.sum(&:suppressed),
|
||||
failed: results.sum(&:failed),
|
||||
errors: results.flat_map(&:errors))
|
||||
end
|
||||
|
||||
def initialize source
|
||||
@source = source
|
||||
end
|
||||
|
||||
def sync!
|
||||
result = Result.new(imported: 0, updated: 0, unchanged: 0,
|
||||
suppressed: 0, failed: 0, errors: [])
|
||||
|
||||
if @source.source_kind == 'google_drive_path'
|
||||
sync_google_drive_path!(result)
|
||||
else
|
||||
candidates.each do |candidate|
|
||||
next if candidate.blank?
|
||||
|
||||
sync_candidate!(candidate, result)
|
||||
end
|
||||
end
|
||||
|
||||
@source.update!(last_synced_at: Time.current)
|
||||
log_result(result)
|
||||
result
|
||||
rescue NotImplementedError, StandardError => e
|
||||
result.failed += 1
|
||||
result.errors << { source_id: @source.id, error: e.message }
|
||||
log_result(result)
|
||||
result
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def candidates
|
||||
case @source.source_kind
|
||||
when 'uri'
|
||||
[uri_candidate]
|
||||
when 'google_drive_file'
|
||||
[google_drive_file_candidate]
|
||||
when 'legacy_drive_path'
|
||||
raise NotImplementedError, 'legacy_drive_path material sync is not implemented'
|
||||
else
|
||||
raise NotImplementedError, "Unsupported material sync source_kind: #{ @source.source_kind }"
|
||||
end
|
||||
end
|
||||
|
||||
def uri_candidate
|
||||
{ source_kind: @source.source_kind,
|
||||
source_uri: @source.source_uri,
|
||||
source_path: @source.source_path,
|
||||
source_file_id: @source.source_file_id,
|
||||
url: @source.source_uri,
|
||||
tag: nil,
|
||||
created_by_user: @source.created_by_user,
|
||||
updated_by_user: @source.updated_by_user || @source.created_by_user }
|
||||
end
|
||||
|
||||
def sync_candidate! candidate, result
|
||||
block = MaterialImportBlockMatcher.match_for_sha256(candidate[:file_sha256])
|
||||
if block
|
||||
result.suppressed += 1
|
||||
Rails.logger.info(
|
||||
material_sync_log(action: 'suppressed',
|
||||
reason: block.reason,
|
||||
normalized_source_key: candidate_normalized_source_key(candidate)))
|
||||
return
|
||||
end
|
||||
|
||||
suppression = suppression_for(candidate)
|
||||
|
||||
if suppression
|
||||
result.suppressed += 1
|
||||
Rails.logger.info(
|
||||
material_sync_log(action: 'suppressed',
|
||||
normalized_source_key: suppression.normalized_source_key))
|
||||
return
|
||||
end
|
||||
|
||||
import = MaterialSyncImporter.import!(candidate)
|
||||
result.public_send("#{ import.action }=", result.public_send(import.action) + 1)
|
||||
rescue StandardError => e
|
||||
result.failed += 1
|
||||
result.errors << { source_id: @source.id,
|
||||
normalized_source_key: candidate_normalized_source_key(candidate),
|
||||
error: e.message }
|
||||
Rails.logger.warn(
|
||||
material_sync_log(action: 'failed',
|
||||
error: e.message,
|
||||
normalized_source_key: candidate_normalized_source_key(candidate)))
|
||||
ensure
|
||||
close_candidate_tempfile(candidate)
|
||||
end
|
||||
|
||||
def google_drive_path_candidates
|
||||
folder_id = google_drive_folder_id
|
||||
Enumerator.new do |entries|
|
||||
drive_client.each_material_file_under_folder(folder_id).each do |entry|
|
||||
entries << build_google_drive_candidate(entry)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def sync_google_drive_path! result
|
||||
folder_id = google_drive_folder_id
|
||||
scanned_count = 0
|
||||
|
||||
drive_client.each_material_file_under_folder(folder_id) do |entry|
|
||||
scanned_count += 1
|
||||
sync_candidate!(build_google_drive_candidate(entry), result)
|
||||
log_google_drive_progress(folder_id, scanned_count, result) if progress_log_scan_count?(scanned_count)
|
||||
end
|
||||
|
||||
log_google_drive_progress(folder_id, scanned_count, result, summary: true)
|
||||
end
|
||||
|
||||
def google_drive_file_candidate
|
||||
entry = drive_client.fetch_material_file(google_drive_file_id)
|
||||
return nil unless entry
|
||||
|
||||
build_google_drive_candidate(entry, single_file: true)
|
||||
end
|
||||
|
||||
def build_google_drive_candidate entry, single_file: false
|
||||
relative_path =
|
||||
if single_file
|
||||
nil
|
||||
else
|
||||
entry[:relative_path]
|
||||
end
|
||||
export_path =
|
||||
MaterialSyncExportPath.build(prefix: @source.export_path_prefix,
|
||||
relative_path:,
|
||||
filename: entry[:name],
|
||||
source_file_id: entry[:id])
|
||||
|
||||
{ source_kind: 'google_drive_file',
|
||||
source_uri: entry[:web_view_link] || google_drive_file_url(entry[:id]),
|
||||
source_path: relative_path.presence || MaterialSyncExportPath.normalize_path(entry[:name]),
|
||||
source_file_id: entry[:id],
|
||||
filename: entry[:name],
|
||||
content_type: entry[:mime_type],
|
||||
file_downloader: lambda {
|
||||
drive_client.download_to_tempfile(entry[:id], filename: entry[:name])
|
||||
},
|
||||
file_sha256: entry[:sha256_checksum],
|
||||
export_path:,
|
||||
profile: @source.profile,
|
||||
tag: nil,
|
||||
url: nil,
|
||||
created_by_user: @source.created_by_user,
|
||||
updated_by_user: @source.updated_by_user || @source.created_by_user }
|
||||
end
|
||||
|
||||
def candidate_normalized_source_key candidate
|
||||
MaterialSyncSuppression.normalize_source_key(source_kind: candidate[:source_kind],
|
||||
source_uri: candidate[:source_uri],
|
||||
source_path: candidate[:source_path],
|
||||
source_file_id: candidate[:source_file_id])
|
||||
end
|
||||
|
||||
def suppression_for candidate
|
||||
if candidate[:source_kind] == 'google_drive_file' && candidate[:source_path].present?
|
||||
return MaterialSyncSuppressionMatcher.match_google_drive_candidate(
|
||||
drive_file_id: candidate[:source_file_id],
|
||||
relative_path: candidate[:source_path])
|
||||
end
|
||||
|
||||
MaterialSyncSuppressionMatcher.match(source_kind: candidate[:source_kind],
|
||||
source_uri: candidate[:source_uri],
|
||||
source_path: candidate[:source_path],
|
||||
source_file_id: candidate[:source_file_id])
|
||||
end
|
||||
|
||||
def drive_client
|
||||
@drive_client ||= GoogleDrive::ApiClient.new
|
||||
end
|
||||
|
||||
def google_drive_folder_id
|
||||
google_drive_target_id.tap do |id|
|
||||
if id.blank?
|
||||
raise ArgumentError, 'google_drive_path source_file_id or source_uri is required'
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def google_drive_file_id
|
||||
google_drive_target_id.tap do |id|
|
||||
if id.blank?
|
||||
raise ArgumentError, 'google_drive_file source_file_id or source_uri is required'
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def google_drive_target_id
|
||||
@source.source_file_id.presence || drive_client.extract_file_id(@source.source_uri)
|
||||
end
|
||||
|
||||
def google_drive_file_url file_id
|
||||
"https://drive.google.com/file/d/#{ file_id }/view"
|
||||
end
|
||||
|
||||
def close_candidate_tempfile candidate
|
||||
tempfile = candidate[:file_tempfile]
|
||||
return unless tempfile
|
||||
|
||||
tempfile.close! unless tempfile.closed?
|
||||
rescue StandardError
|
||||
nil
|
||||
end
|
||||
|
||||
def log_result result
|
||||
Rails.logger.info(material_sync_log(imported: result.imported,
|
||||
updated: result.updated,
|
||||
unchanged: result.unchanged,
|
||||
suppressed: result.suppressed,
|
||||
failed: result.failed))
|
||||
end
|
||||
|
||||
def progress_log_scan_count? scanned_count
|
||||
scanned_count == 1 || (scanned_count % 50).zero?
|
||||
end
|
||||
|
||||
def log_google_drive_progress folder_id, scanned_count, result, summary: false
|
||||
Rails.logger.info(
|
||||
material_sync_log(folder_id:,
|
||||
scanned_count:,
|
||||
imported: result.imported,
|
||||
updated: result.updated,
|
||||
unchanged: result.unchanged,
|
||||
suppressed: result.suppressed,
|
||||
failed: result.failed,
|
||||
progress: summary ? 'summary' : 'scan'))
|
||||
end
|
||||
|
||||
def material_sync_log fields
|
||||
{ material_sync_source_id: @source.id,
|
||||
material_sync_source_name: @source.name }.merge(fields).to_json
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,68 @@
|
||||
class MaterialSyncSuppressionMatcher
|
||||
def self.match source_kind:,
|
||||
source_uri: nil,
|
||||
drive_path: nil,
|
||||
drive_file_id: nil,
|
||||
source_path: nil,
|
||||
source_file_id: nil
|
||||
key = MaterialSyncSuppression.normalize_source_key(source_kind:,
|
||||
source_uri:,
|
||||
drive_path:,
|
||||
drive_file_id:,
|
||||
source_path:,
|
||||
source_file_id:)
|
||||
return nil if key.blank?
|
||||
|
||||
MaterialSyncSuppression.find_by(normalized_source_key: key)
|
||||
end
|
||||
|
||||
def self.match_google_drive_candidate drive_file_id:, relative_path:
|
||||
normalized_path = validate_relative_path!(relative_path)
|
||||
|
||||
exact = [
|
||||
normalize_key('google_drive_file', drive_file_id),
|
||||
normalize_key('google_drive_path', normalized_path),
|
||||
normalize_key('legacy_drive_path', normalized_path),
|
||||
].compact
|
||||
|
||||
suppression =
|
||||
MaterialSyncSuppression.find_by(normalized_source_key: exact)
|
||||
return suppression if suppression
|
||||
|
||||
MaterialSyncSuppression
|
||||
.where(source_kind: ['google_drive_path_prefix', 'legacy_drive_path_prefix'])
|
||||
.where('drive_path = :path OR :path LIKE CONCAT(drive_path, "/%")', path: normalized_path)
|
||||
.order(:id)
|
||||
.first
|
||||
end
|
||||
|
||||
def self.suppressed?(...)
|
||||
match(...).present?
|
||||
end
|
||||
|
||||
def self.validate_relative_path! relative_path
|
||||
raw = relative_path.to_s.delete("\0").tr('\\', '/').strip
|
||||
raise ArgumentError, 'relative_path is required' if raw.blank?
|
||||
raise ArgumentError, 'relative_path must be relative' if raw.start_with?('/')
|
||||
raise ArgumentError, 'relative_path must be relative' if raw.match?(/\A[A-Za-z]:\//)
|
||||
if raw.start_with?('My Drive/', 'マイドライブ/')
|
||||
raise ArgumentError, 'relative_path must be relative to source folder'
|
||||
end
|
||||
raise ArgumentError, 'relative_path must not contain //' if raw.include?('//')
|
||||
raise ArgumentError, 'relative_path must not end with /' if raw.end_with?('/')
|
||||
|
||||
parts = raw.split('/')
|
||||
if parts.any? { |part| part.in?(['.', '..']) }
|
||||
raise ArgumentError, 'relative_path must not contain dot segments'
|
||||
end
|
||||
|
||||
raw
|
||||
end
|
||||
|
||||
def self.normalize_key source_kind, value
|
||||
return nil if value.blank?
|
||||
|
||||
MaterialSyncSuppression.normalize_source_key(source_kind:, source_path: value,
|
||||
source_file_id: value)
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,51 @@
|
||||
class MaterialSyncSuppressionRegistrar
|
||||
def self.create! attributes, created_by_user:
|
||||
new(attributes, created_by_user:).create!
|
||||
end
|
||||
|
||||
def initialize attributes, created_by_user:
|
||||
@attributes = attributes
|
||||
@created_by_user = created_by_user
|
||||
end
|
||||
|
||||
def create!
|
||||
suppression = nil
|
||||
|
||||
MaterialSyncSuppression.transaction do
|
||||
suppression = MaterialSyncSuppression.create!(
|
||||
@attributes.merge(created_by_user: @created_by_user))
|
||||
discard_existing_materials!(suppression)
|
||||
end
|
||||
|
||||
suppression
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def discard_existing_materials! suppression
|
||||
matching_materials(suppression).find_each do |material|
|
||||
MaterialVersionRecorder.ensure_snapshot!(material, created_by_user: @created_by_user)
|
||||
material.discard!
|
||||
MaterialVersionRecorder.record!(material:,
|
||||
event_type: :discard,
|
||||
created_by_user: @created_by_user)
|
||||
end
|
||||
end
|
||||
|
||||
def matching_materials suppression
|
||||
materials = Material.unscoped.kept
|
||||
|
||||
case suppression.source_kind
|
||||
when 'google_drive_path', 'legacy_drive_path'
|
||||
materials.where(source_path: suppression.drive_path)
|
||||
when 'google_drive_path_prefix', 'legacy_drive_path_prefix'
|
||||
path = suppression.drive_path.to_s
|
||||
materials.where(
|
||||
'source_path = :path OR source_path LIKE :prefix',
|
||||
path:,
|
||||
prefix: "#{ path }/%")
|
||||
else
|
||||
materials.where(normalized_source_key: suppression.normalized_source_key)
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,126 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
require 'mini_magick'
|
||||
require 'open3'
|
||||
require 'tempfile'
|
||||
|
||||
|
||||
class MaterialThumbnailGenerator
|
||||
SIZE = '180x180'
|
||||
|
||||
class << self
|
||||
def generate! material
|
||||
new(material).generate!
|
||||
end
|
||||
end
|
||||
|
||||
def initialize material
|
||||
@material = material
|
||||
@ffmpeg_stderr = []
|
||||
end
|
||||
|
||||
def generate!
|
||||
return log_result(:no_file) unless @material.file.attached?
|
||||
return log_result(:unsupported_content_type) unless image? || video?
|
||||
|
||||
@material.file.blob.open do |file|
|
||||
thumbnail = image? ? image_thumbnail(file.path) : video_thumbnail(file.path)
|
||||
return log_result(:generation_failed) unless thumbnail
|
||||
|
||||
return attach_thumbnail(thumbnail)
|
||||
end
|
||||
rescue ActiveStorage::FileNotFoundError => e
|
||||
log_result(:file_not_found, error: e)
|
||||
rescue MiniMagick::Error => e
|
||||
log_result(:mini_magick_error, error: e)
|
||||
rescue ArgumentError, StandardError => e
|
||||
log_result(:generation_failed, error: e)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def image? = content_type.start_with?('image/')
|
||||
|
||||
def video? = content_type.start_with?('video/')
|
||||
|
||||
def content_type
|
||||
return nil unless @material.file.attached?
|
||||
|
||||
@material.file.blob.content_type.to_s
|
||||
end
|
||||
|
||||
def image_thumbnail path
|
||||
image = MiniMagick::Image.open(path)
|
||||
image.resize(SIZE)
|
||||
image.format('jpg')
|
||||
image
|
||||
end
|
||||
|
||||
def video_thumbnail path
|
||||
[1, 0].each do |seconds|
|
||||
tempfile = Tempfile.new(['material-thumbnail', '.jpg'])
|
||||
tempfile.close
|
||||
ok = extract_video_frame(path, tempfile.path, seconds)
|
||||
next unless ok && File.size?(tempfile.path)
|
||||
|
||||
return image_thumbnail(tempfile.path)
|
||||
ensure
|
||||
tempfile&.unlink
|
||||
end
|
||||
|
||||
nil
|
||||
end
|
||||
|
||||
def extract_video_frame input_path, output_path, seconds
|
||||
_stdout, stderr, status =
|
||||
Open3.capture3('ffmpeg',
|
||||
'-y',
|
||||
'-ss', seconds.to_s,
|
||||
'-i', input_path,
|
||||
'-frames:v', '1',
|
||||
'-f', 'image2',
|
||||
output_path)
|
||||
@ffmpeg_stderr << stderr if stderr.present?
|
||||
status.success?
|
||||
rescue Errno::ENOENT => e
|
||||
@ffmpeg_stderr << "ffmpeg unavailable: #{ e.message }"
|
||||
false
|
||||
end
|
||||
|
||||
def attach_thumbnail image
|
||||
blob = nil
|
||||
File.open(image.path) do |io|
|
||||
blob = ActiveStorage::Blob.create_and_upload!(
|
||||
io:,
|
||||
filename: 'material-thumbnail.jpg',
|
||||
content_type: 'image/jpeg')
|
||||
end
|
||||
@material.thumbnail.attach(blob)
|
||||
log_result(:attached)
|
||||
rescue StandardError => e
|
||||
blob&.purge_later
|
||||
log_result(:attach_failed, error: e)
|
||||
end
|
||||
|
||||
def log_result result, error: nil
|
||||
log_payload = { material_id: @material.id,
|
||||
file_blob_id: file_blob_id,
|
||||
content_type:,
|
||||
result:,
|
||||
error_class: error&.class&.name,
|
||||
error_message: error&.message,
|
||||
ffmpeg_stderr: @ffmpeg_stderr.join("\n").presence }
|
||||
if [:attached, :no_file, :unsupported_content_type].include?(result)
|
||||
Rails.logger.info("Material thumbnail generation: #{ log_payload.to_json }")
|
||||
else
|
||||
Rails.logger.warn("Material thumbnail generation: #{ log_payload.to_json }")
|
||||
end
|
||||
result
|
||||
end
|
||||
|
||||
def file_blob_id
|
||||
return nil unless @material.file.attached?
|
||||
|
||||
@material.file.blob.id
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,73 @@
|
||||
class MaterialVersionRecorder < VersionRecorder
|
||||
EVENT_TYPES = ['create', 'update', 'discard', 'restore'].freeze
|
||||
|
||||
def self.record! material:, event_type:, created_by_user:, file_snapshot: nil
|
||||
new(material:, event_type:, created_by_user:, file_snapshot:).record!
|
||||
end
|
||||
|
||||
def initialize material:, event_type:, created_by_user:, file_snapshot: nil
|
||||
@file_snapshot = file_snapshot
|
||||
|
||||
super(record: material, event_type:, created_by_user:)
|
||||
end
|
||||
|
||||
def self.ensure_snapshot! material, created_by_user:
|
||||
return if material.material_versions.exists?
|
||||
|
||||
record!(material:, event_type: :create,
|
||||
created_by_user: material.created_by_user || created_by_user)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def version_class = MaterialVersion
|
||||
def version_association = :material_versions
|
||||
def record_key = :material
|
||||
|
||||
def snapshot_attributes
|
||||
blob = @record.file.attached? ? @record.file.blob : nil
|
||||
file_snapshot = build_file_snapshot(blob)
|
||||
|
||||
{ url: @record.url,
|
||||
parent: @record.parent,
|
||||
tag: @record.tag,
|
||||
tag_name: @record.tag&.name,
|
||||
tag_category: @record.tag&.category,
|
||||
source_kind: @record.source_kind,
|
||||
source_uri: @record.source_uri,
|
||||
source_path: @record.source_path,
|
||||
source_file_id: @record.source_file_id,
|
||||
normalized_source_key: @record.normalized_source_key,
|
||||
export_paths_json: @record.snapshot_export_paths,
|
||||
discarded_at: @record.discarded_at,
|
||||
file_blob_id: file_snapshot[:file_blob_id],
|
||||
file_filename: file_snapshot[:file_filename],
|
||||
file_content_type: file_snapshot[:file_content_type],
|
||||
file_byte_size: file_snapshot[:file_byte_size],
|
||||
file_checksum: file_snapshot[:file_checksum],
|
||||
file_sha256: file_snapshot[:file_sha256] }
|
||||
end
|
||||
|
||||
def build_file_snapshot blob
|
||||
return @file_snapshot if @file_snapshot
|
||||
return empty_file_snapshot unless blob
|
||||
|
||||
{ file_blob_id: blob.id,
|
||||
file_filename: blob.filename.to_s,
|
||||
file_content_type: blob.content_type,
|
||||
file_byte_size: blob.byte_size,
|
||||
file_checksum: blob.checksum,
|
||||
file_sha256: MaterialFileSha256.metadata_sha256(blob) }
|
||||
end
|
||||
|
||||
def empty_file_snapshot
|
||||
{ file_blob_id: nil,
|
||||
file_filename: nil,
|
||||
file_content_type: nil,
|
||||
file_byte_size: nil,
|
||||
file_checksum: nil,
|
||||
file_sha256: nil }
|
||||
end
|
||||
|
||||
def event_types = self.class::EVENT_TYPES
|
||||
end
|
||||
@@ -0,0 +1,148 @@
|
||||
require 'stringio'
|
||||
require 'zlib'
|
||||
|
||||
# Initial implementation keeps every file payload and the final ZIP in memory.
|
||||
# Keep this service boundary stable so job/cached export paths can replace it later.
|
||||
class MaterialZipExporter
|
||||
Entry = Struct.new(:path, :data, :mtime, keyword_init: true)
|
||||
MissingFile = Struct.new(:material_id, :export_path, :blob_id, :filename, keyword_init: true)
|
||||
|
||||
class EmptyExportError < StandardError; end
|
||||
class DuplicatePathError < StandardError; end
|
||||
class MissingFileError < StandardError
|
||||
attr_reader :missing_files
|
||||
|
||||
def initialize missing_files
|
||||
@missing_files = missing_files
|
||||
|
||||
super("Missing files: #{missing_files.map(&:export_path).join(', ')}")
|
||||
end
|
||||
end
|
||||
|
||||
def initialize profile: 'legacy_drive', tag_id: nil
|
||||
@profile = profile.presence || 'legacy_drive'
|
||||
@tag_id = tag_id.presence
|
||||
end
|
||||
|
||||
def export
|
||||
entries = build_entries
|
||||
raise EmptyExportError if entries.empty?
|
||||
|
||||
ZipWriter.write(entries)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def build_entries
|
||||
rows = MaterialExportItem
|
||||
.enabled
|
||||
.includes(material: { file_attachment: :blob })
|
||||
.joins(:material)
|
||||
.merge(Material.kept)
|
||||
.where(profile: @profile)
|
||||
.order(:export_path)
|
||||
|
||||
rows = rows.where(materials: { tag_id: @tag_id }) if @tag_id
|
||||
|
||||
missing_files = []
|
||||
|
||||
entries = rows.filter_map do |item|
|
||||
material = item.material
|
||||
next unless material.file.attached?
|
||||
|
||||
data = download_blob(item, missing_files)
|
||||
next unless data
|
||||
|
||||
Entry.new(path: item.export_path,
|
||||
data:,
|
||||
mtime: material.updated_at || Time.current)
|
||||
end
|
||||
|
||||
raise MissingFileError.new(missing_files) if missing_files.any?
|
||||
|
||||
paths = entries.map(&:path)
|
||||
duplicated = paths.find { |path| paths.count(path) > 1 }
|
||||
raise DuplicatePathError, duplicated if duplicated
|
||||
|
||||
entries
|
||||
end
|
||||
|
||||
def download_blob item, missing_files
|
||||
blob = item.material.file.blob
|
||||
|
||||
blob.download
|
||||
rescue ActiveStorage::FileNotFoundError
|
||||
missing_files << MissingFile.new(
|
||||
material_id: item.material_id,
|
||||
export_path: item.export_path,
|
||||
blob_id: blob.id,
|
||||
filename: blob.filename.to_s,
|
||||
)
|
||||
nil
|
||||
end
|
||||
|
||||
class ZipWriter
|
||||
VERSION_NEEDED = 20
|
||||
GP_FLAG = 0x0800
|
||||
COMPRESSION_STORE = 0
|
||||
|
||||
def self.write entries
|
||||
new(entries).write
|
||||
end
|
||||
|
||||
def initialize entries
|
||||
@entries = entries
|
||||
@central_directory = []
|
||||
end
|
||||
|
||||
def write
|
||||
io = StringIO.new(''.b)
|
||||
|
||||
@entries.each do |entry|
|
||||
write_entry(io, entry)
|
||||
end
|
||||
|
||||
central_start = io.pos
|
||||
@central_directory.each { |header| io.write(header) }
|
||||
central_size = io.pos - central_start
|
||||
|
||||
io.write([0x06054b50, 0, 0, @entries.size, @entries.size,
|
||||
central_size, central_start, 0].pack('VvvvvVVv'))
|
||||
io.string
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def write_entry io, entry
|
||||
path = entry.path.b
|
||||
data = entry.data.b
|
||||
crc32 = Zlib.crc32(data)
|
||||
dos_time, dos_date = dos_timestamp(entry.mtime)
|
||||
offset = io.pos
|
||||
|
||||
local_header = [0x04034b50, VERSION_NEEDED, GP_FLAG, COMPRESSION_STORE,
|
||||
dos_time, dos_date, crc32, data.bytesize, data.bytesize,
|
||||
path.bytesize, 0].pack('VvvvvvVVVvv')
|
||||
|
||||
io.write(local_header)
|
||||
io.write(path)
|
||||
io.write(data)
|
||||
|
||||
@central_directory << central_header(path:, crc32:, size: data.bytesize,
|
||||
dos_time:, dos_date:, offset:)
|
||||
end
|
||||
|
||||
def central_header path:, crc32:, size:, dos_time:, dos_date:, offset:
|
||||
[0x02014b50, VERSION_NEEDED, VERSION_NEEDED, GP_FLAG, COMPRESSION_STORE,
|
||||
dos_time, dos_date, crc32, size, size, path.bytesize, 0, 0, 0, 0, 0,
|
||||
offset].pack('VvvvvvvVVVvvvvvVV') + path
|
||||
end
|
||||
|
||||
def dos_timestamp time
|
||||
local = time.to_time
|
||||
dos_time = (local.hour << 11) | (local.min << 5) | (local.sec / 2)
|
||||
dos_date = ((local.year - 1980) << 9) | (local.month << 5) | local.day
|
||||
[dos_time, dos_date]
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -73,7 +73,7 @@ class VersionRecorder
|
||||
end
|
||||
|
||||
def validate_event_type!
|
||||
return if EVENT_TYPES.include?(@event_type)
|
||||
return if event_types.include?(@event_type)
|
||||
|
||||
raise ArgumentError, "Invalid event_type: #{ @event_type }"
|
||||
end
|
||||
@@ -84,4 +84,5 @@ class VersionRecorder
|
||||
def snapshot_attributes = raise NotImplementedError
|
||||
|
||||
def record_class = @record.class
|
||||
def event_types = self.class::EVENT_TYPES
|
||||
end
|
||||
|
||||
新しい課題から参照
ユーザをブロックする