コミットを比較

..

146 コミット

作成者 SHA1 メッセージ 日付
みてるぞ e0debed94e #399 2026-07-18 17:31:57 +09:00
みてるぞ 583ce22a7e #399 2026-07-18 17:07:56 +09:00
みてるぞ 7c808a6f76 #399 2026-07-18 15:15:10 +09:00
みてるぞ a5ae7c6f2d #399 2026-07-18 14:55:11 +09:00
みてるぞ 03dc4d0661 #399 2026-07-18 13:32:22 +09:00
みてるぞ 06b9c1cb50 #399 2026-07-18 13:07:49 +09:00
みてるぞ 688b4af575 #399 2026-07-18 12:53:50 +09:00
みてるぞ 040cc3f25d #399 2026-07-18 12:15:32 +09:00
みてるぞ b3e67d8cca #399 2026-07-18 11:55:09 +09:00
みてるぞ b00a62a0ac #399 2026-07-18 09:11:11 +09:00
みてるぞ e6c3c635b8 #399 2026-07-18 02:29:55 +09:00
みてるぞ 9552081133 #399 2026-07-18 02:08:20 +09:00
みてるぞ 2d2a9b4bd6 #399 2026-07-18 01:37:11 +09:00
みてるぞ 09ac2576bb #399 2026-07-18 01:28:22 +09:00
みてるぞ 9eca670934 #399 2026-07-18 01:12:55 +09:00
みてるぞ c31d84115d #399 2026-07-18 00:31:02 +09:00
みてるぞ ff970f8171 #399 2026-07-18 00:01:12 +09:00
みてるぞ 0ae41b6266 #399 2026-07-17 23:14:37 +09:00
みてるぞ eb4bf5e35c #399 2026-07-17 22:21:18 +09:00
みてるぞ e6b7e33b83 #399 2026-07-17 21:53:24 +09:00
みてるぞ 3820d3d4d5 #399 2026-07-17 21:06:26 +09:00
みてるぞ dd2d199d04 #399 2026-07-16 23:39:13 +09:00
みてるぞ 0a8ffc38b8 #399 2026-07-16 23:23:32 +09:00
みてるぞ 3f75994bd4 #399 2026-07-16 23:12:14 +09:00
みてるぞ f91b78bd47 #399 2026-07-16 22:31:32 +09:00
みてるぞ 7f8cce39bc #399 2026-07-16 22:05:40 +09:00
みてるぞ 5c5a9fa1b0 #399 2026-07-16 22:00:16 +09:00
みてるぞ 66c738cbff #399 2026-07-16 21:45:02 +09:00
みてるぞ cde0a2deae #399 2026-07-16 20:54:42 +09:00
みてるぞ 43a3772976 #399 2026-07-16 20:35:07 +09:00
みてるぞ 90d8d3ff08 #399 2026-07-16 20:05:09 +09:00
みてるぞ d1de631eed #399 2026-07-16 19:35:21 +09:00
みてるぞ 6f3ccb0ffa #399 2026-07-16 19:18:35 +09:00
みてるぞ 95c3f08a44 #399 2026-07-16 18:52:07 +09:00
みてるぞ ef95b20a7e #399 2026-07-16 18:33:45 +09:00
みてるぞ f76fbe6711 #399 2026-07-16 18:16:20 +09:00
みてるぞ 0224c4d2f4 #399 2026-07-16 12:39:35 +09:00
みてるぞ ead825e3ee #399 2026-07-16 07:56:45 +09:00
みてるぞ 3d88fdecb1 #399 2026-07-16 01:19:36 +09:00
みてるぞ 2e1b4449ba #399 2026-07-16 00:44:11 +09:00
みてるぞ 4eb483c049 #399 2026-07-16 00:24:24 +09:00
みてるぞ e197f27e9a #399 2026-07-15 23:59:01 +09:00
みてるぞ 34f81325d2 #399 2026-07-15 23:42:05 +09:00
みてるぞ 9af4670c6c #399 2026-07-15 23:29:19 +09:00
みてるぞ 5183b0b354 #399 2026-07-15 23:06:28 +09:00
みてるぞ d0914fce3b #399 2026-07-15 22:51:01 +09:00
みてるぞ 0ac7332458 #399 2026-07-15 22:41:06 +09:00
みてるぞ 07ce19e32d #399 2026-07-15 22:26:58 +09:00
みてるぞ 5f1d619139 #399 2026-07-15 22:04:47 +09:00
みてるぞ 3cd22855b3 #399 2026-07-15 21:42:00 +09:00
みてるぞ 55c69866a7 #399 2026-07-15 21:37:03 +09:00
みてるぞ 04d2f0f5a9 #399 2026-07-15 20:59:00 +09:00
みてるぞ 025f49cbcb #399 2026-07-15 20:51:24 +09:00
みてるぞ d2f1b1ed48 #399 2026-07-15 20:25:51 +09:00
みてるぞ 6e5aa1e30f #399 2026-07-15 19:57:09 +09:00
みてるぞ 21d6c1408a #399 2026-07-15 19:43:39 +09:00
みてるぞ eebbd5d069 #399 2026-07-15 19:33:18 +09:00
みてるぞ 58828597d7 #399 2026-07-15 08:03:25 +09:00
みてるぞ b0c24f319a #399 2026-07-14 20:03:11 +09:00
みてるぞ f9463f383f #399 2026-07-14 19:37:04 +09:00
みてるぞ 6d037192c4 #399 2026-07-14 19:12:23 +09:00
みてるぞ d035da99ad #399 2026-07-14 18:39:24 +09:00
みてるぞ bc660676ef Merge remote-tracking branch 'origin/main' into feature/399 2026-07-14 12:40:32 +09:00
みてるぞ 38650b5671 URL 正規化 (#409) (#410)
Reviewed-on: #410
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-07-14 12:38:02 +09:00
みてるぞ 97b132e5ab #399 2026-07-13 01:31:45 +09:00
みてるぞ be68841bd3 #399 2026-07-13 01:17:11 +09:00
みてるぞ ac82adc6b3 #399 2026-07-13 00:58:02 +09:00
みてるぞ aa96ec95d1 #399 2026-07-12 14:03:05 +09:00
みてるぞ df17f20907 #399 2026-07-12 13:14:36 +09:00
みてるぞ 769966648b #399 2026-07-12 12:43:31 +09:00
みてるぞ 7bcb76516c #399 2026-07-12 12:23:53 +09:00
みてるぞ 8970edc59f #399 2026-07-12 11:17:23 +09:00
みてるぞ 9b1ea56e36 #399 2026-07-12 10:27:13 +09:00
みてるぞ 155edfe018 #399 2026-07-12 02:22:36 +09:00
みてるぞ 23f1ffb04b #399 2026-07-12 02:08:32 +09:00
みてるぞ ccfe65a6a4 #399 2026-07-12 02:01:25 +09:00
みてるぞ 06d6e512e4 #399 2026-07-12 01:56:24 +09:00
みてるぞ 5bd097bcfe #399 2026-07-12 01:47:34 +09:00
みてるぞ f636d2a177 #399 2026-07-12 01:39:31 +09:00
みてるぞ 10dc776313 #399 2026-07-12 01:17:45 +09:00
みてるぞ a9e16735f8 #399 2026-07-12 01:11:07 +09:00
みてるぞ 440a6c9961 #399 2026-07-12 00:54:21 +09:00
みてるぞ 4535a9d260 #399 2026-07-12 00:46:53 +09:00
みてるぞ 440d3d38be #399 2026-07-11 23:21:59 +09:00
みてるぞ cb33d9ca25 #399 2026-07-11 23:08:03 +09:00
みてるぞ 19bf24432a #399 2026-07-11 23:07:51 +09:00
みてるぞ ac41385962 Merge remote-tracking branch 'origin/main' into feature/399 2026-07-11 22:36:03 +09:00
みてるぞ 08bf92ff79 #399 2026-07-11 21:46:10 +09:00
みてるぞ c51d7b98ba #399 2026-07-11 21:11:36 +09:00
みてるぞ b7b284c076 スマホでのメニュー表示デザイン・バグ (#405) (#408)
Reviewed-on: #408
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-07-11 05:02:23 +09:00
みてるぞ 6c451d260f サムネ取得 API 見直し (#398) (#402)
Reviewed-on: #402
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-07-10 01:27:11 +09:00
みてるぞ c1902fbc99 素材管理で新規タグ紐づけ時にタグ履歴が発生しない問題 (#386) (#392)
開発環境で動作確認したらマージする.

Reviewed-on: #392
2026-07-09 06:43:06 +09:00
みてるぞ 2f2b6e2afa タグ D&D 時の表示位置修正 (#268) (#400)
Reviewed-on: #400
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-07-08 00:01:07 +09:00
みてるぞ f5b632ed89 設定画面 (#34) (#397)
Reviewed-on: #397
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-07-07 00:48:41 +09:00
みてるぞ 15619886f2 サイド・バーの幅を可変に (#385) (#396)
Reviewed-on: #396
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-07-05 03:38:01 +09:00
みてるぞ fdf652951a 【再掲】タグの局所記載 (#351) (#380)
#353 を再開できなかったので.

Reviewed-on: #380
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-07-03 03:23:36 +09:00
みてるぞ f96ddb65f1 タグが見切れる問題対応 (#76) (#395)
Reviewed-on: #395
2026-07-02 23:40:01 +09:00
みてるぞ 5657fa7e7b “開発モード” の表示 (#389) (#393)
Reviewed-on: #393
2026-07-02 12:47:19 +09:00
みてるぞ 19a185d5b5 上映会タグ表示バグ修正 (#360) (#384)
まだ画面チェックしてゐないのでマージ禁止

Reviewed-on: #384
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-07-02 01:37:49 +09:00
みてるぞ acd062f289 タグ循環の禁止 (#332) (#382)
Reviewed-on: #382
2026-06-30 01:20:55 +09:00
みてるぞ 877876b661 サムネつきで広場に追加する際にエラーとなる問題修正 (#352) (#390)
Reviewed-on: #390
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-06-30 01:19:23 +09:00
みてるぞ bffd4c422a 過去のタグ名に名称変更できないバグ修正 (#383) (#387)
Reviewed-on: #387
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-06-30 01:18:38 +09:00
みてるぞ 776dea87d9 素材管理 (#306) (#381)
開発環境では、**DB を壊さない前提で、migration → API → 画面 → 同期 → ZIP → 履歴**の順に見るのがよいです。今回の差分は素材管理全体に触っているので、単体でチョンチョン見るより、素材の一生を通すのが早いです。

## 0. 先に方針

**やらないこと:**

```sh id="snb3i6"
rails db:drop
rails db:reset
rails db:setup
DISABLE_DATABASE_ENVIRONMENT_CHECK=1 ...
```

これは禁止。
開発 DB に本番データを入れているなら、床板を剥がして耐震確認するようなものです。

---

## 1. migration 確認

まず現在の状態を見る。

```sh id="qsdfpt"
cd backend
RAILS_ENV=development bundle exec rails db:migrate:status
```

その後、通常 migration。

```sh id="a89fir"
RAILS_ENV=development bundle exec rails db:migrate
```

見るポイント:

```txt id="c1u57a"
materials に source_* / normalized_source_key / version_no がある
material_versions に event_type / file snapshot / source snapshot がある
material_export_items がある
material_sync_suppressions がある
material_sync_sources がある
既存 materials に material_versions version_no=1 create が backfill されている
```

確認用:

```sh id="g4vd4m"
RAILS_ENV=development bundle exec rails runner '
puts "materials=#{Material.count}"
puts "versions=#{MaterialVersion.count}"
puts "materials without versions=#{Material.left_joins(:material_versions).where(material_versions: { id: nil }).count}"
puts "sync suppressions table=#{ActiveRecord::Base.connection.table_exists?(:material_sync_suppressions)}"
'
```

ここで `materials without versions=0` になれば、backfill は通っています。

---

## 2. 既存素材一覧の画面確認

フロントを起動して `/materials` を見る。

```sh id="vl28jd"
cd frontend
npm run dev
```

見る観点:

```txt id="i2ovxw"
初期表示で素材が出る
初期表示ではグルーピングがオフ
タグなし素材も出る
カード表示でサムネまたは代替テキストが出る
一覧表示に切り替えられる
q / tag_state / media_kind / sort / direction が効く
```

ここでまず、普通の素材一覧が壊れていないことを確認します。

---

## 3. 左タグバーの確認

`/materials` を開いて左タグバーからタグを選ぶ。

見る観点:

```txt id="n6udul"
URL が tag_id=...&include_descendants=1&group_by=parent_tag になる
選択中タグが左バーで強調される
一覧上部に「選択中」の表示が出る
「タグ選択を解除」で通常表示に戻れる
解除後、tag_id / include_descendants / group_by / page が消える
子タグ・孫タグの素材も一覧に出る
親タググルーピングされる
```

特に重要なのはこれ。

```txt id="d33os0"
親タグ A を選択
  A に直接紐づく素材
  A > B に紐づく素材
  A > B > C に紐づく素材
が同じ一覧に出ること
```

---

## 4. 選択タグから素材追加

左タグからタグを選択した状態で、一覧上部の **このタグに素材を追加** を押す。

見る観点:

```txt id="qdd5uw"
素材追加画面の tag 欄に選択中タグ名が初期入力されている
file または url を指定して保存できる
保存後 return_to で元のタグ選択済み一覧に戻る
戻った一覧に追加した素材が出る
material_versions に create が 1 件できる
```

Rails console でも確認できます。

```sh id="i07rrb"
RAILS_ENV=development bundle exec rails runner '
m = Material.order(id: :desc).first
puts({ id: m.id, tag: m.tag&.name, versions: m.material_versions.count, version_no: m.version_no }.inspect)
'
```

---

## 5. グループ見出しから素材追加

親タググルーピング表示中に、各グループ見出しの **このタグに素材を追加** を押す。

見る観点:

```txt id="fpx8w4"
グループタグ名が tag 欄に初期入力される
保存後、元の一覧に戻る
追加素材がそのグループ内に出る
```

ここは今回の導線の肝です。棚の見出しから直接その棚へ素材を置けるかを見る。

---

## 6. 素材更新と履歴

既存素材の詳細または編集導線から、タグ・URL・ファイル・export path を更新する。

見る観点:

```txt id="w36i61"
更新前 snapshot が無ければ create が補われる
更新後 update version ができる
file_blob_id / file_filename / file_sha256 が material_versions に入る
export_paths_json が履歴に残る
/materials/changes または /materials/versions で履歴が見える
```

console:

```sh id="a3okhv"
RAILS_ENV=development bundle exec rails runner '
m = Material.order(updated_at: :desc).first
puts m.material_versions.order(:version_no).map { |v|
  [v.version_no, v.event_type, v.tag_name, v.file_filename, v.file_sha256, v.export_paths_hash]
}.inspect
'
```

---

## 7. サムネイル

画像素材を追加して、一覧にサムネイルが出るか確認。

動画素材があるなら、`ffmpeg` が入っている環境で backfill。

```sh id="qxm8fj"
cd backend
RAILS_ENV=development bundle exec rails materials:thumbnails:backfill
```

見る観点:

```txt id="s9a1vf"
画像は 180x180 のサムネが付く
動画はフレームからサムネが作られる
非対応ファイルは代替テキスト表示になる
ログに result が出る
```

---

## 8. ZIP export

export path がある素材を用意して、ブラウザで確認。

```txt id="aq7f7o"
/materials/download.zip?profile=legacy_drive
```

見る観点:

```txt id="h4rve5"
ZIP が落ちる
entry path が material_export_items.export_path になる
disabled な export item は入らない
ファイル実体が欠けている場合は 422 と missing_files が返る
```

---

## 9. 抑止

`/materials/suppressions` で path prefix 抑止を追加する。

見る観点:

```txt id="w80jbu"
member で作成できる
guest は forbidden / unauthorized
google_drive_path_prefix で既存素材が discard される
discard 履歴が material_versions に残る
同期時に同じ source_path 配下が再作成されない
```

---

## 10. Google Drive 同期

開発環境では、まず小さいフォルダでやるのがよいです。
いきなり本番素材集フォルダを食わせると、ログが藪になります 🌿

必要な ENV:

```sh id="kzbb7f"
GOOGLE_DRIVE_SERVICE_ACCOUNT_EMAIL=...
GOOGLE_DRIVE_PRIVATE_KEY_PATH=...
MATERIAL_SYNC_SOURCE_KIND=google_drive_path
MATERIAL_SYNC_SOURCE_FILE_ID=<folder_id>
MATERIAL_SYNC_SOURCE_NAME=dev-small-folder
MATERIAL_SYNC_SOURCE_PROFILE=legacy_drive
```

seed で source 作成、または console で作成。

```sh id="sec40c"
RAILS_ENV=development bundle exec rails db:seed
RAILS_ENV=development bundle exec rails materials:sync
```

見る観点:

```txt id="a5ltqu"
imported / updated / unchanged / suppressed / failed がログに出る
2 回目実行で unchanged が増える
tag は nil のままでも保存できる
人手で tag / url を付けた既存同期素材が、再同期で消えない
Google native file は skip される
download 後 sha256 block が効く
```

---

## 11. schema.rb は別途確認

これはテストというより merge gate です。
今回まだ怪しいので、差分に以下が混ざっていないことを確認します。

```txt id="s3k6da"
wiki_assets 削除
wiki_pages.next_asset_no 削除
素材管理と無関係な CHECK constraint 削除
素材管理と無関係な index order 消失
```

ここが残るなら、機能テストが通っても merge は止めた方がいいです。

---

## 最小テスト順

時間がないなら、この順で十分です。

```txt id="c1k2pw"
1. db:migrate
2. materials without versions = 0 を確認
3. /materials 初期表示
4. 左タグ選択 → 子孫込み表示 → グルーピング
5. タグ選択解除
6. 選択タグから素材追加 → return_to で戻る
7. グループ見出しから素材追加
8. 更新して material_versions を確認
9. ZIP export
10. 小さい Drive folder で materials:sync を 2 回
```

これで、今回の差分の主要な導線はほぼ踏めます。

Reviewed-on: #381
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-06-28 06:35:53 +09:00
みてるぞ ec2b3d2254 タグ “廃止” 追加 (#378) (#379)
Reviewed-on: #379
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-06-22 08:40:06 +09:00
みてるぞ ffd28c0f9e グカネータ スコア補正 (#376) (#377)
Reviewed-on: #377
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-06-18 01:15:54 +09:00
みてるぞ a54ca72244 グカネータ改良 (#371) (#375)
Reviewed-on: #375
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-06-17 01:04:57 +09:00
みてるぞ 5bbd6eda11 グカネータ軽量モード廃止 (#370) (#372)
Reviewed-on: #372
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-06-15 22:14:08 +09:00
みてるぞ ece95838f0 グカネータ公開 / 洗澡鹿のパス変更 (#361) (#369)
Reviewed-on: #369
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-06-14 05:40:31 +09:00
みてるぞ 7ab46f907f グカネータ公開 (#361) (#368)
Reviewed-on: #368
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-06-14 05:33:39 +09:00
みてるぞ e94720941c グカネータ作成 / ウィニング・ラン修正 (#41) (#366)
Reviewed-on: #366
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-06-12 02:08:59 +09:00
みてるぞ def6870f06 グカネータ / 質問パターン見直し (#41) (#365)
Reviewed-on: #365
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-06-12 01:35:31 +09:00
みてるぞ c361c561c2 グカネータ作成 / 質問パターン修正 (#41) (#364)
Reviewed-on: #364
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-06-11 23:21:44 +09:00
みてるぞ 979ccf598e グカネータ作成 / テスト型バグ修正 (#41) (#363)
Reviewed-on: #363
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-06-10 23:43:50 +09:00
みてるぞ 37ade2a988 グカネータ作成 (#041) (#362)
Reviewed-on: #362
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-06-10 23:33:56 +09:00
みてるぞ 7d48a8f694 上映会ニコニコ・バグ修正 (#358) (#359)
Reviewed-on: #359
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-06-07 09:08:41 +09:00
みてるぞ 3980e9651e 上映会改修 (#302) (#357)
Reviewed-on: #357
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-06-07 02:51:25 +09:00
みてるぞ 750aa40e8e フォームのバリデーションとニコ連携の画面変更 (#090) (#355)
Reviewed-on: #355
Co-authored-by: miteruzo <miteruzo@naver.com>
Co-committed-by: miteruzo <miteruzo@naver.com>
2026-06-05 01:59:46 +09:00
みてるぞ dc54f9cbb5 Merge pull request 'フロントのテスト追加 (#155)' (#350) from feature/155 into main
Reviewed-on: #350
2026-05-13 22:03:05 +09:00
みてるぞ 78143363c9 #155 2026-05-13 21:49:40 +09:00
みてるぞ 0a13c00f37 #155 2026-05-13 20:42:25 +09:00
みてるぞ add60cb413 #155 2026-05-11 03:32:47 +09:00
みてるぞ fb761b199d さらに修正 (#346) (#349)
Merge remote-tracking branch 'origin/main' into feature/346

#346

#346

Merge remote-tracking branch 'origin/main' into feature/346

#346

#346

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #349
2026-05-11 02:46:08 +09:00
みてるぞ 73152f2934 ちょっと修正 (#346) (#348)
#346

Merge remote-tracking branch 'origin/main' into feature/346

#346

#346

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #348
2026-05-11 02:43:08 +09:00
みてるぞ 2de7e13a8a Codex 用ファイル追加 (#346) (#347)
#346

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #347
2026-05-11 02:30:55 +09:00
みてるぞ e03cc01109 投稿排他 (#171) (#345)
#171

#171

#171

#171

#171

#171

#171

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #345
2026-05-10 11:16:49 +09:00
みてるぞ b47cdc7ad7 BAN の実装 (#327) (#342)
#327

#327

#327

#327

Merge remote-tracking branch 'origin/main' into feature/327

#327

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #342
2026-05-04 16:22:13 +09:00
みてるぞ 52aa1615b6 ニジラー詳細ページ作成 (#63) (#341)
#63

#63

#63

#63

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #341
2026-05-04 03:37:12 +09:00
みてるぞ dceed1caa1 親投稿機能 (#46) (#339)
Merge remote-tracking branch 'origin/main' into feature/046

#46

#46

#46

#46

#46

#46

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #339
2026-05-03 03:21:35 +09:00
みてるぞ 5002859fc8 YouTube の自動同期 (#314) (#340)
#314

#314

#314

#314

#314

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #340
2026-05-02 17:56:14 +09:00
みてるぞ fcd3b87b2a 奪はれた別名の履歴追加 (#329) (#338)
#329

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #338
2026-04-27 12:45:06 +09:00
みてるぞ 0ff7fdf78a Wiki のバージョン管理 (#317) (#333)
#317

#317

#317

#317

#317

#317

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #333
2026-04-26 22:17:25 +09:00
みてるぞ b2c3e02ccc ユーザ作成時に IP アドレス連携するやぅに (#323) (#326)
Merge branch 'main' into feature/323

Merge branch 'main' into feature/323

Merge branch 'main' into feature/323

#323

#323

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #326
2026-04-25 21:00:22 +09:00
みてるぞ c112576b11 ニコニコ DB 逆連携 (#200) (#331)
#200

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #331
2026-04-25 20:46:49 +09:00
みてるぞ 6235b293f0 タグ履歴ページ (#321) (#330)
#321

#321

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #330
2026-04-24 02:21:26 +09:00
みてるぞ 43cd38a216 タグ詳細 (#318) (#328)
#318

#318

#318

#318

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #328
2026-04-23 00:06:49 +09:00
みてるぞ 8ff1819d5a 同期バグ修正 (#324) (#325)
#324

Reviewed-on: #325
2026-04-19 23:04:15 +09:00
みてるぞ bde7d33949 タグ履歴 (#309) (#319)
#309

#309

#309

#309

#309

Merge remote-tracking branch 'origin/main' into feature/309

#309

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #319
2026-04-19 20:21:51 +09:00
みてるぞ 5c7580d571 ニコタグ連携バグ修正 (#294) (#316)
#294

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #316
2026-04-19 16:44:20 +09:00
みてるぞ 48f823a7c8 履歴画面変更(#308) (#315)
Merge branch 'main' into feature/308

#308

#308

#308

#308

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #315
2026-04-18 05:43:33 +09:00
みてるぞ bd11e37fd3 緊急対応(#312) (#313)
'backend/config/storage.yml' を更新

Reviewed-on: #313
2026-04-15 20:21:51 +09:00
みてるぞ e72ec608f4 利用規約(#95) (#311)
#95

#95

#95

#95

#95

Merge remote-tracking branch 'origin/main' into feature/095

#95

#95

#95

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #311
2026-04-14 12:31:48 +09:00
みてるぞ a3914fb22a posts 履歴管理(RSpec 修正)(#264) (#310)
Merge remote-tracking branch 'origin/main' into feature/264

Merge branch 'feature/264' of https://git.miteruzo.com/miteruzo/btrc-hub into feature/264

#264

Merge branch 'main' into feature/264

#264

#264

#264

#264

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #310
2026-04-11 22:13:19 +09:00
みてるぞ c36b2c8a1b 投稿に対する履歴(#264) (#307)
Merge branch 'main' into feature/264

#264

#264

#264

#264

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #307
2026-04-11 17:05:57 +09:00
みてるぞ e021423904 スマホ・レーアウト・バグ(#304) (#305)
#304

#304

#304

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #305
2026-04-11 16:58:41 +09:00
みてるぞ 7b15cb2c5a 素材管理(#99) (#303)
#99

#99

#99

#99

#99

#99

#99

#99

#99

#99

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #303
2026-04-07 07:44:50 +09:00
みてるぞ 2adff3966a 上映会にコメント機能追加(#297) (#299)
#297

#297

#297

#297

#297

Merge remote-tracking branch 'origin/main' into feature/297

#297

#297

#297

#297

#297

#297

#297

#297

#297

Co-authored-by: miteruzo <miteruzo@naver.com>
Reviewed-on: #299
2026-03-26 23:01:54 +09:00
465個のファイルの変更59627行の追加3981行の削除
+35
ファイルの表示
@@ -0,0 +1,35 @@
## 背景
なぜ必要か。
## 対象範囲
- backend:
- frontend:
- docs:
- migration:
## やること
- [ ]
## 受け入れ条件
- [ ]
## 実行すべき確認
- [ ] `cd backend && bundle exec rspec`
- [ ] `cd frontend && npm run build`
- [ ] `cd frontend && npm run lint`
## 禁止事項
- unrelated refactor はしない
- 既存 API response shape を壊さない
- 認証・認可・BAN を弱めない
## Codex への指示
この issue を読んで実装してください。
不明点があれば、実装前に調査結果と選択肢を提示してください。
+1318
ファイルの表示
ファイル差分が大きすぎるため省略します 差分を読込み
+411
ファイルの表示
@@ -0,0 +1,411 @@
# backend/AGENTS.md
## Scope
These rules apply to work under `backend/`.
This is a Rails API app using Active Record, RSpec, request specs,
service objects, representation classes, and version tables for post/tag/wiki
history.
## Commands
Use commands backed by files and dependencies in this directory:
```sh
bin/setup
bin/dev
bin/rails
bin/rake
bin/rubocop
bin/brakeman
bundle exec rspec
```
Common checks:
```sh
bundle exec rspec
bin/rubocop
bin/brakeman
```
Common Rails commands:
```sh
bin/rails db:prepare
bin/rails db:migrate
bin/rails routes
bin/rails server
```
After backend behavior changes, run the relevant RSpec files. For broad backend changes, run:
```sh
bundle exec rspec
```
If a command cannot be run or fails, report the exact command and failure.
Do not create, modify, or run tests unless the user explicitly asks for test
work. When the user asks for tests, keep working and rerun them until they
pass or the remaining failure is clearly blocked.
## Rails structure
- `app/controllers`: API controllers.
- `app/models`: Active Record models and concerns.
- `app/representations`: JSON response shaping.
- `app/services`: domain services such as version recorders, wiki commit,
YouTube sync, and similarity calculation.
- `config/routes.rb`: public API routes.
- `db/migrate`: migrations.
- `db/schema.rb`: schema snapshot.
- `lib/tasks`: custom Rake tasks.
- `spec`: RSpec tests.
Before changing behavior, inspect the matching route, controller, model,
service, representation, and spec.
## Shared backend systems
Before adding backend behaviour, search the existing backend first. At minimum,
check these locations:
- `app/controllers`
- `app/controllers/concerns`
- `app/models`
- `app/models/concerns`
- `app/representations`
- `app/services`
- `app/services/*`
- `app/jobs`
- `lib`
- `lib/tasks`
- `config/initializers`
Do not infer commonality from directory names alone. Read the actual
responsibility and representative usage sites.
### Controller reuse
Before adding logic to a controller, inspect:
- `ApplicationController` authentication, authorization, BAN, and IP BAN
- existing render and validation-error helpers
- existing param parsing
- controller concerns
- the controller for the same resource
- existing services
- existing representations
Keep controllers focused on:
- authentication and authorization
- parameter intake
- service and model invocation
- HTTP status selection
- representation selection
Do not reimplement these per controller when an existing path already owns
them:
- authentication and role checks
- validation error JSON
- URL normalisation
- tag normalisation
- thumbnail handling
- version recording
- complex transactions
- external HTTP fetching
- response representation assembly
### Authentication, authorization, and BAN
Treat these as the canonical backend entrypoints:
- `ApplicationController#authenticate_user`
- `current_user`
- `X-Transfer-Code`
- `reject_banned_ip_address!`
- `reject_banned_user!`
- `gte_member?`
- `admin?`
Do not create feature-local permission services, role comparisons, or header
parsing when the existing authentication boundary already owns the behaviour.
If the current boundary is insufficient, extend it minimally instead of adding
another permission path.
### Representations
If an endpoint for the same resource already uses `app/representations`, do not
assemble a separate JSON shape directly inside the controller without first
checking the existing representation contract.
Inspect at least:
- `PostRepr`
- `TagRepr`
- `MaterialRepr`
- `TheatreRepr`
- `UserRepr`
- `WikiPageRepr`
- `DeerjikistRepr`
When a lightweight response is genuinely different in purpose, keep it
deliberate and compatible with the surrounding contracts. Do not force every
identifier list into a large representation, but do not fork the same resource
shape casually either.
### Domain services
When work touches multiple models, transactions, external APIs, file handling,
history creation, or multi-step workflow, search `app/services` first.
At minimum, search for existing services in these responsibility areas:
- version recorder and versioning
- wiki commit
- YouTube or Google Drive API client
- material sync or ZIP export
- similarity calculation
- theatre selection or skip finalisation
- metadata, thumbnail, or file processing
- URL normaliser or sanitisation
- import or export
- preview safety or HTTP fetch
Do not create a same-responsibility service under another namespace or another
name. If an existing service is close, extend that API minimally instead of
wrapping it in a feature-local service.
### Versioning
When a feature writes history, snapshots, or restore roots, search the existing
versioning path first. At minimum, inspect:
- `VersionRecorder`
- `PostVersionRecorder`
- `TagVersionRecorder`
- `TagVersioning`
- `MaterialVersionRecorder`
- `NicoTagVersionRecorder`
- `WikiVersionRecorder`
Do not implement history writes in controllers, callbacks, or ad hoc feature
services when the recorder layer already owns the transaction boundary and
meaning.
### Normalisation, sanitisation, and parsing
For URLs, tag names, times, video durations, identifiers, and paths, search the
existing normaliser, sanitisation rule, parser, and model-callback path first.
Do not let frontend, controller, service, and model each invent different rules
for the same value. Use one canonical normalisation path and keep input
validation distinct from pre-persistence normalisation.
### External HTTP and URL safety
When fetching external URLs, reuse the existing preview-safety stack. Search at
least for:
- URL safety
- redirect validation
- response size limits
- timeouts
- network failure mapping
- HTML metadata extraction
- known-site extraction
- thumbnail fetching
Do not add direct `Net::HTTP`, `Faraday`, or equivalent feature-local HTTP code
that reimplements SSRF checks, redirect restrictions, size limits, or timeouts.
If the current fetcher is insufficient, extend its existing safety contract.
### Storage, files, and Active Storage
When handling files, thumbnails, ZIP output, object storage, or Active Storage
blobs, inspect existing storage helpers, exporters, thumbnail generators, and
checksum helpers first. Do not reimplement the same attach, export path,
download, resize, or checksum flow in a controller or one-off service.
### Concerns
Do not create controller or model concerns merely because some code is shared.
Use a concern only when multiple classes share the same lifecycle, macro,
callback, or tightly cohesive behaviour. Utility collections belong in explicit
objects or services, not in `CommonConcern`, `SharedMethods`, or `Utils`.
### Model boundaries
Model-specific invariants, associations, validations, and normalisation may
live in the model. Multi-model workflow, external access, complex transaction
flow, and feature orchestration belong in services. Do not hide feature
workflow in model callbacks.
### Transactions, locking, and race handling
If transactions, locking, idempotency, or race recovery already exist in a
service or model method, do not add a second implementation in a controller or
new service. Inspect the existing transaction boundary first, avoid wrapping
the same operation in needless nested transactions, and handle unique-constraint
races according to the target constraint's business meaning.
## Ruby style
- Prefer precise, minimal changes.
- Use single quotes unless interpolation or escaping makes double quotes better.
- Do not put a space before Ruby method-call parentheses.
- For `render`-family method calls, omit parentheses even when passing
keyword arguments.
- Never put a line break immediately before `)` in Ruby.
- Do not use `%w` or `%i` in new Ruby code.
- Never write a Ruby line longer than 99 characters.
- Aim to keep Ruby lines within 79 characters where practical.
- For small Ruby method definitions that take keyword arguments, match the
local no-parentheses style when nearby code uses it.
- When an `if` condition is split across multiple lines and combines clauses
with `&&` or `||`, wrap the whole condition in parentheses.
- Treat Ruby hash `{ ... }` style and Ruby block `{ ... }` style as separate
rules.
- Do not format Ruby hashes like Ruby blocks.
- For Ruby hashes, keep the closing `}` on the same line as the final pair.
- Keep the first pair on the same line as `{` by default.
- Short Ruby hashes may stay visually compact across two lines with the first
pair kept on the opening line and aligned continuation pairs below it.
- If the hash would exceed the line limit, break after `{` and indent pairs
by 4 spaces.
- Put one logical pair per line when the expression would otherwise become
dense.
- For Ruby arrays, never put whitespace or a line break immediately before `]`.
- Keep the first element on the same line as `[` by default.
- If an array would exceed the line limit, break after `[` and indent
elements by 4 spaces.
- For Ruby blocks, use 2-space indentation for the block body.
- Keep comments short and useful; avoid narrating obvious code.
- Do not add production dependencies without approval.
## Authentication and authorization
- Authentication is handled through the `X-Transfer-Code` header in
`ApplicationController#authenticate_user`.
- `current_user` is set by looking up `User.inheritance_code`.
- Do not bypass or weaken the `X-Transfer-Code` flow unless the task
explicitly changes authentication.
- Unauthenticated write actions should return `:unauthorized` consistently
with existing controllers.
- Role checks use `User` enum roles: `guest`, `member`, and `admin`.
- Use `current_user.gte_member?` for member-or-admin write permissions where
existing controllers do so.
- Use `current_user.admin?` only for admin-only paths, such as tag child relationship changes.
- Do not replace role checks with looser presence checks.
## BAN and IP BAN
- `ApplicationController` runs these before actions in order:
- `reject_banned_ip_address!`
- `authenticate_user`
- `reject_banned_user!`
- User and IP bans use `banned_at`, not a boolean `banned` column.
- `User#banned?` and `IpAddress#banned?` check `banned_at.present?`.
- Do not weaken BAN or IP BAN behavior.
- If changing request authentication or controller before actions, add or
update request specs covering banned users and banned IP addresses only when
the user explicitly asks for tests.
## RSpec
- Prefer RSpec for new backend tests.
- Put API behavior coverage under `spec/requests`.
- Put model behavior under `spec/models`.
- Put service behavior under `spec/services`.
- Put Rake task coverage under `spec/tasks`.
- `spec/rails_helper.rb` loads `spec/support/**/*.rb`.
- Request specs include `AuthHelper` and `JsonHelper`.
- `AuthHelper#sign_in_as(user)` stubs
`ApplicationController#current_user`; use it when matching existing
request spec style.
- Add or update request specs for API behavior changes only when the user
explicitly asks for tests, especially status codes, permissions, response
shape, and version conflict behavior.
## Migrations
- Keep migrations and `db/schema.rb` consistent.
- Use reversible migrations where practical; otherwise define explicit `up` and `down`.
- For data backfills inside migrations, follow the existing pattern of
defining migration-local `ActiveRecord::Base` classes with
`self.table_name`.
- Preserve existing indexes, foreign keys, check constraints, and null constraints.
- Be careful with MySQL-specific options already present in migrations, such as `after:`.
- Do not edit old migrations just to change current behavior unless
explicitly requested; add a new migration.
## Version tables
- Versioned records include posts, tags, nico tags, and wiki pages.
- Current records have `version_no`; version tables have positive
`version_no` with unique indexes scoped to the parent record.
- Version event types are `create`, `update`, `discard`, and `restore`.
- Version rows are readonly through the `VersionRecord` concern.
- Use the existing recorder services instead of manually inserting version
rows in application code:
- `PostVersionRecorder`
- `TagVersionRecorder`
- `NicoTagVersionRecorder`
- `WikiVersionRecorder`
- `TagVersioning`
- `VersionRecorder` locks the current record, validates sequence consistency,
skips unchanged update snapshots, creates the next version row, and updates
the record `version_no`.
- Do not update versioned records without considering whether a version snapshot must be created.
- For optimistic concurrency paths, preserve `base_version_no`, `force`, and
`merge` semantics. Cover conflicts in request specs only when the user
explicitly asks for tests.
## Domain cautions
- Posts have tag snapshots, parent post implications, original-created ranges,
viewed state, and version conflict behavior.
- Tags have canonical names, aliases through `TagName`, categories, parent
implications, discard behavior, and version snapshots.
- Nico tags have separate relation/version behavior; do not treat them like
normal editable tags without checking existing code.
- Wiki pages involve page content, revisions/history, version rows,
title/tag-name behavior, and diff/restore paths.
- Materials, theatres, and comments have user and permission checks; inspect
the controller before changing them.
## API responses
- Use representation classes under `app/representations` when existing endpoints do.
- Keep response keys consistent with existing JSON contracts.
- Frontend code expects camelCase conversion client-side, while Rails params
and JSON keys are generally snake_case.
- Preserve existing HTTP status conventions:
`:unauthorized` for no user, `:forbidden` for insufficient role or banned
user, `:not_found` for missing records, and `:unprocessable_entity` for
validation failures.
- For diagnostic or internal helper JSON, prefer a deliberately light response
shape over full representation classes when callers only need identifiers,
labels, URLs, or weights.
## Active Record performance
- When a controller action serializes nested associations, preload the
associations it will touch instead of allowing N+1 queries.
- Be sensitive to N+1 queries in all backend work.
- Avoid introducing N+1 queries, and proactively fix existing N+1 issues when
you find them in the code path you are editing.
- When an association may already be preloaded, prefer loaded-association
checks that reuse the preloaded data without losing the efficient database
path.
## Files to avoid in routine work
- Do not inspect or edit `tmp/`, `log/`, `storage/`, `vendor/`, or dependency
directories unless explicitly needed.
- Do not modify generated schema or migration output without the corresponding
migration when schema changes are made.
+4 -2
ファイルの表示
@@ -50,8 +50,6 @@ group :development, :test do
gem 'factory_bot_rails'
end
gem "mysql2", "~> 0.5.6"
gem "image_processing", "~> 1.14"
@@ -69,3 +67,7 @@ gem 'whenever', require: false
gem 'discard'
gem "rspec-rails", "~> 8.0", :groups => [:development, :test]
gem 'aws-sdk-s3', require: false
gem 'rails-i18n', '~> 8.0.0'
+25
ファイルの表示
@@ -73,6 +73,25 @@ GEM
tzinfo (~> 2.0, >= 2.0.5)
uri (>= 0.13.1)
ast (2.4.3)
aws-eventstream (1.4.0)
aws-partitions (1.1231.0)
aws-sdk-core (3.244.0)
aws-eventstream (~> 1, >= 1.3.0)
aws-partitions (~> 1, >= 1.992.0)
aws-sigv4 (~> 1.9)
base64
bigdecimal
jmespath (~> 1, >= 1.6.1)
logger
aws-sdk-kms (1.123.0)
aws-sdk-core (~> 3, >= 3.244.0)
aws-sigv4 (~> 1.5)
aws-sdk-s3 (1.217.0)
aws-sdk-core (~> 3, >= 3.244.0)
aws-sdk-kms (~> 1)
aws-sigv4 (~> 1.5)
aws-sigv4 (1.12.1)
aws-eventstream (~> 1, >= 1.0.2)
base64 (0.2.0)
bcrypt_pbkdf (1.1.1)
bcrypt_pbkdf (1.1.1-arm64-darwin)
@@ -157,6 +176,7 @@ GEM
pp (>= 0.6.0)
rdoc (>= 4.0.0)
reline (>= 0.4.2)
jmespath (1.6.2)
json (2.12.0)
jwt (2.10.1)
base64
@@ -286,6 +306,9 @@ GEM
rails-html-sanitizer (1.6.2)
loofah (~> 2.21)
nokogiri (>= 1.15.7, != 1.16.7, != 1.16.6, != 1.16.5, != 1.16.4, != 1.16.3, != 1.16.2, != 1.16.1, != 1.16.0.rc1, != 1.16.0)
rails-i18n (8.0.2)
i18n (>= 0.7, < 2)
railties (>= 8.0.0, < 9)
railties (8.0.2)
actionpack (= 8.0.2)
activesupport (= 8.0.2)
@@ -441,6 +464,7 @@ PLATFORMS
x86_64-linux-musl
DEPENDENCIES
aws-sdk-s3
bootsnap
brakeman
diff-lcs
@@ -456,6 +480,7 @@ DEPENDENCIES
puma (>= 5.0)
rack-cors
rails (~> 8.0.2)
rails-i18n (~> 8.0.0)
rspec-rails (~> 8.0)
rubocop-rails-omakase
sprockets-rails
+67 -4
ファイルの表示
@@ -1,14 +1,19 @@
class ApplicationController < ActionController::API
before_action :authenticate_user
rescue_from ActiveRecord::RecordInvalid, with: :render_record_invalid
rescue_from ActiveRecord::RecordNotUnique, with: :render_record_not_unique
def current_user
@current_user
end
before_action :reject_banned_ip_address!
before_action :authenticate_user
before_action :reject_banned_user!
def current_user = @current_user
private
def authenticate_user
code = request.headers['X-Transfer-Code'] || request.headers['HTTP_X_TRANSFER_CODE']
return if code.blank?
@current_user = User.find_by(inheritance_code: code)
end
@@ -22,4 +27,62 @@ class ApplicationController < ActionController::API
s.in?(['', '1', 'true', 'on', 'yes'])
end
end
def render_bad_request message = 'リクエストが不正です.'
render json: { type: 'bad_request',
message:,
errors: { },
base_errors: [message] },
status: :bad_request
end
def render_unprocessable_entity message = '入力を確認してください.', field: nil
render_validation_error(fields: field ? { field => [message] } : { },
base: field ? [] : [message])
end
def render_record_invalid error
render_validation_error error.record
end
def render_record_not_unique _error = nil
render_validation_error base: ['すでに存在してゐます.']
end
def reject_banned_ip_address!
ip_address = IpAddress.find_by(ip_address: IPAddr.new(request.remote_ip).hton)
return unless ip_address&.banned?
head :forbidden
end
def reject_banned_user!
return unless current_user&.banned?
head :forbidden
end
def render_validation_error record = nil, fields: { }, base: [], status: :unprocessable_entity
errors = { }
if record
record.errors.each do |error|
errors[error.attribute] ||= []
errors[error.attribute] << error.message
end
end
fields.each do |attr, messages|
errors[attr.to_sym] ||= []
errors[attr.to_sym].concat(Array(messages))
end
base_errors = Array(base) + Array(errors.delete(:base))
render json: { type: 'validation_error',
message: '入力内容を確認してください.',
errors:,
base_errors: },
status:
end
end
+7 -3
ファイルの表示
@@ -2,7 +2,8 @@ class DeerjikistsController < ApplicationController
def show
platform = params[:platform].to_s.strip
code = params[:code].to_s.strip
return head :bad_request if platform.blank? || code.blank?
return render_bad_request('platform は必須です.') if platform.blank?
return render_bad_request('code は必須です.') if code.blank?
deerjikist = Deerjikist
.joins(:tag)
@@ -22,7 +23,9 @@ class DeerjikistsController < ApplicationController
platform = params[:platform].to_s.strip
code = params[:code].to_s.strip
tag_id = params[:tag_id].to_i
return head :bad_request if platform.blank? || code.blank? || tag_id <= 0
return render_bad_request('platform は必須です.') if platform.blank?
return render_bad_request('code は必須です.') if code.blank?
return render_bad_request('tag_id が不正です.') if tag_id <= 0
deerjikist = Deerjikist.find_or_initialize_by(platform:, code:).tap do |d|
d.tag_id = tag_id
@@ -38,7 +41,8 @@ class DeerjikistsController < ApplicationController
platform = params[:platform].to_s.strip
code = params[:code].to_s.strip
return head :bad_request if platform.blank? || code.blank?
return render_bad_request('platform は必須です.') if platform.blank?
return render_bad_request('code は必須です.') if code.blank?
Deerjikist.find([platform, code]).destroy!
+272
ファイルの表示
@@ -0,0 +1,272 @@
class GekanatorGamesController < ApplicationController
def create
return head :unauthorized unless current_user
guessed_post_id = params.require(:guessed_post_id)
correct_post_id = params[:correct_post_id].presence
answers = params.require(:answers).as_json
game = GekanatorGame.new(
user: current_user,
guessed_post_id:,
correct_post_id:,
won: correct_post_id.present? && guessed_post_id.to_i == correct_post_id.to_i,
question_count: answers.length,
answers:)
if game.invalid?
render json: { errors: game.errors.full_messages }, status: :unprocessable_entity
return
end
learned_example_count = 0
ActiveRecord::Base.transaction do
game.save!
learned_example_count = learn_answers_from_game!(game)
end
render json: {
id: game.id,
learned_example_count:
}, status: :created
rescue ActiveRecord::RecordInvalid => e
render json: { errors: e.record.errors.full_messages }, status: :unprocessable_entity
end
def extra_questions
game = find_owned_game
return if performed?
questions =
GekanatorQuestion
.accepted
.includes(:gekanator_question_examples)
.where(kind: 'post_similarity', source: 'user_suggested')
.to_a
selected =
prioritized_extra_questions(
questions,
post_id: game.correct_post_id,
user: current_user,
limit: 6)
render json: {
questions: selected.map { |question| extra_question_json(question) }
}
end
def extra_question_answers
game = find_owned_game
return if performed?
answer_params = params.require(:answers)
if !answer_params.is_a?(Array)
return render_validation_error fields: { answers: ['配列で指定してください.'] }
end
answers = answer_params.map { |answer|
{
question_id: answer.require(:question_id).to_i,
answer: answer.require(:answer)
}
}
questions = GekanatorQuestion.where(id: answers.map { _1[:question_id] })
question_by_id = questions.index_by(&:id)
if questions.length != answers.length
return render_validation_error fields: { answers: ['質問が見つかりません.'] }
end
if questions.any? { |question| question.status != 'accepted' || question.kind != 'post_similarity' }
return render_validation_error fields: { answers: ['質問が不正です.'] }
end
ActiveRecord::Base.transaction do
answers.each do |item|
question = question_by_id[item[:question_id]]
example =
GekanatorQuestionExample.find_or_initialize_by(
gekanator_question: question,
post: game.correct_post,
user: current_user)
example.record_answer!(
answer: item[:answer],
source: 'post_game_extra',
gekanator_game: game)
example.save!
end
end
render json: { count: answers.length }, status: :created
end
private
def extra_question_json question
{
id: question.id,
text: question.text,
source: question.source,
priority_weight: question.priority_weight
}
end
def prioritized_extra_questions questions, post_id:, user:, limit:
answered_question_ids =
GekanatorQuestionExample
.where(user:, gekanator_question_id: questions.map(&:id))
.distinct
.pluck(:gekanator_question_id)
unanswered, answered =
questions.partition { |question| !answered_question_ids.include?(question.id) }
selected = weighted_sample_questions(unanswered, post_id:, limit:)
return selected if selected.length >= limit
selected + weighted_sample_questions(
answered.reject { |question| selected.any? { _1.id == question.id } },
post_id:,
limit: limit - selected.length)
end
def weighted_sample_questions questions, post_id:, limit:
remaining = questions.uniq(&:id)
selected = []
while selected.length < limit && remaining.any?
weighted =
remaining.map { |question|
[question, selection_weight_for(question, post_id: post_id)]
}
total_weight = weighted.sum { |_question, weight| weight }
break if total_weight <= 0
target = rand * total_weight
cumulative = 0.0
chosen =
weighted.find do |_question, weight|
cumulative += weight
cumulative >= target
end&.first || weighted.first.first
selected << chosen
remaining.reject! { |question| question.id == chosen.id }
end
selected
end
def selection_weight_for question, post_id:
sample_count =
question.gekanator_question_examples.sum { |example|
next 0 unless example.post_id == post_id
example.sample_count.presence || 1
}
question.priority_weight.to_f / (1.0 + sample_count * 0.15)
end
def find_owned_game
return head :unauthorized unless current_user
game = GekanatorGame.find_by(id: params[:id])
return head :not_found unless game
if !current_user.admin? && game.user_id != current_user.id
return head :not_found
end
game
end
def learn_answers_from_game! game
correct_post = game.correct_post
return 0 if correct_post.blank?
accepted_questions =
GekanatorQuestion
.accepted
.index_by { |question| public_question_id_for(question) }
learned_count = 0
Array(game.answers).each do |answer|
answer_value = answer['answer'].to_s
next if answer_value.blank? || answer_value == 'unknown'
question_id = game_answer_question_id(answer)
next if question_id.blank?
question = accepted_questions[question_id.to_s]
next unless learnable_game_answer_question?(question)
example =
GekanatorQuestionExample.find_or_initialize_by(
gekanator_question: question,
post: correct_post,
user: current_user)
example.record_answer!(
answer: answer_value,
source: 'post_game_answer',
gekanator_game: game)
example.save!
learned_count += 1
end
learned_count
end
def public_question_id_for question
condition = normalize_condition(question.condition)
case condition[:type]
when 'tag'
"tag:#{condition[:key]}"
when 'source'
"source:#{condition[:host]}"
when 'original-year'
"original-year:#{condition[:year]}"
when 'original-month'
"original-month:#{condition[:month]}"
when 'original-month-day'
"original-month-day:#{condition[:monthDay] || condition[:month_day]}"
when 'title-length-at-least'
"title:length-at-least:#{condition[:length]}"
when 'title-length-greater-than'
"title:length-at-least:#{condition[:length].to_i + 1}"
when 'title-has-ascii'
'title:ascii'
when 'title-contains'
"title:contains:#{condition[:text]}"
when 'post-similarity'
"post-similarity:#{question.id}"
else
"catalog:#{question.id}"
end
end
def normalize_condition condition
json = condition.deep_dup.as_json
if json['type'] == 'original-month-day' && json['monthDay'].blank?
json['monthDay'] = json.delete('month_day')
end
json.deep_symbolize_keys
end
def learnable_game_answer_question? question
return false if question.nil?
return true if question.kind == 'post_similarity'
return false unless question.kind == 'tag'
condition = normalize_condition(question.condition)
key = condition[:key].to_s
!key.start_with?('nico:')
end
def game_answer_question_id answer
answer['question_id'] ||
answer[:question_id] ||
answer['questionId'] ||
answer[:questionId]
end
end
+71
ファイルの表示
@@ -0,0 +1,71 @@
class GekanatorPostsController < ApplicationController
def index
posts =
Post
.preload(:post_similarities, tags: :tag_name)
.with_attached_thumbnail
.order(Arel.sql(
'COALESCE(posts.original_created_before - INTERVAL 1 MINUTE, ' \
'posts.original_created_from, posts.created_at) DESC, posts.id DESC'))
.to_a
active_tags_by_post_id =
posts.each_with_object({ }) do |post, h|
h[post.id] = post.tags.reject(&:deprecated?)
end
render json: {
posts: posts.map { |post|
post_json(post,
active_tags_by_post_id:)
}
}
end
private
def post_json post, active_tags_by_post_id:
{
id: post.id,
url: post.url,
title: post.title,
thumbnail: thumbnail_url(post),
thumbnail_base: post.thumbnail_base,
original_created_from: post.original_created_from,
original_created_before: post.original_created_before,
post_similarity_edges: post_similarity_edges_json(
post,
active_tags_by_post_id:),
tags: active_tags_by_post_id.fetch(post.id, []).map { |tag| tag_json(tag) }
}
end
def post_similarity_edges_json post, active_tags_by_post_id:
post
.post_similarities
.filter_map do |similarity|
next unless active_tags_by_post_id.key?(similarity.target_post_id)
{
target_post_id: similarity.target_post_id,
cos: similarity.cos.to_f
}
end
end
def tag_json tag
{
id: tag.id,
name: tag.name,
category: tag.category
}
end
def thumbnail_url post
return nil unless post.thumbnail.attached?
rails_storage_proxy_url(post.thumbnail, only_path: false)
rescue
nil
end
end
+95
ファイルの表示
@@ -0,0 +1,95 @@
class GekanatorQuestionSuggestionsController < ApplicationController
def create
return head :unauthorized unless current_user
game = GekanatorGame.find_by(id: params.require(:gekanator_game_id))
return head :not_found unless game
if !current_user.admin? && game.user_id != current_user.id
return head :not_found
end
existing_question_id = params[:existing_question_id].presence
if existing_question_id
question = GekanatorQuestion.accepted.find_by(id: existing_question_id)
return head :not_found unless question
unless learnable_existing_question?(question)
return render_validation_error fields: { existing_question_id: ['質問が不正です.'] }
end
example =
GekanatorQuestionExample.find_or_initialize_by(
gekanator_question: question,
post: game.correct_post,
user: current_user)
example.record_answer!(
answer: params.require(:answer),
source: 'post_game_extra',
gekanator_game: game)
if example.save
render json: {
id: question.id,
count: game.question_suggestions.count
}, status: :created
else
render_validation_error example
end
return
end
suggestion = GekanatorQuestionSuggestion.new(
gekanator_game: game,
user: current_user,
question_text: params.require(:question_text),
answer: params.require(:answer))
if suggestion.valid?
ActiveRecord::Base.transaction do
suggestion.save!
Gekanator::QuestionSuggestionPromoter.call(
suggestion: suggestion,
user: current_user)
end
render json: {
id: suggestion.id,
count: game.question_suggestions.count
}, status: :created
else
render_validation_error suggestion
end
end
def ai_convert
return head :not_found unless current_user&.admin?
suggestion = GekanatorQuestionSuggestion.find_by(id: params[:id])
return head :not_found unless suggestion
if Gekanator::AiRunBudget.exceeded_after_next_run?
suggestion.gekanator_ai_runs.create!(
model: 'budget_guard',
status: 'blocked_budget',
input_tokens: 0,
output_tokens: 0,
estimated_cost_jpy: 0)
return head :payment_required
end
Gekanator::QuestionSuggestionAiConverter.call(
suggestion: suggestion,
user: current_user)
head :no_content
rescue NotImplementedError
head :not_implemented
end
private
def learnable_existing_question? question
return true if question.kind == 'post_similarity'
return false unless question.kind == 'tag'
key = question.condition.as_json['key'].to_s
!key.start_with?('nico:')
end
end
+147
ファイルの表示
@@ -0,0 +1,147 @@
class GekanatorQuestionsController < ApplicationController
def index
questions =
GekanatorQuestion
.accepted
.includes(:gekanator_question_examples)
.order(priority_weight: :desc, id: :asc)
.to_a
deprecated_tag_keys = deprecated_tag_keys_for(questions)
render json: {
questions: questions.filter_map { |question|
json = question_json(question)
next if hidden_question?(json[:condition], deprecated_tag_keys)
json
}
}
end
private
def question_json question
condition = condition_json(question.condition).deep_symbolize_keys
json = {
record_id: question.id,
id: question_id_for(question, condition),
text: question_text_for(question, condition),
kind: question.kind,
condition: condition,
source: question.source,
priority_weight: question.priority_weight
}
if question.kind == 'post_similarity' || question.kind == 'tag'
json[:example_answers] = example_answers_json(question)
end
json
end
def question_id_for question, condition
case condition[:type]
when 'tag'
"tag:#{ condition[:key] }"
when 'source'
"source:#{ condition[:host] }"
when 'original-year'
"original-year:#{ condition[:year] }"
when 'original-month'
"original-month:#{ condition[:month] }"
when 'original-month-day'
"original-month-day:#{ condition[:monthDay] || condition[:month_day] }"
when 'title-length-at-least'
"title:length-at-least:#{ condition[:length] }"
when 'title-length-greater-than'
"title:length-at-least:#{ condition[:length].to_i + 1 }"
when 'title-has-ascii'
'title:ascii'
when 'title-contains'
"title:contains:#{ condition[:text] }"
when 'post-similarity'
"post-similarity:#{ question.id }"
else
"catalog:#{ question.id }"
end
end
def condition_json condition
json = condition.deep_dup.as_json
if json['type'] == 'original-month-day' && json['monthDay'].blank?
json['monthDay'] = json.delete('month_day')
end
if json['type'] == 'title-length-greater-than'
json['type'] = 'title-length-at-least'
json['length'] = json['length'].to_i + 1
end
json
end
def question_text_for question, condition
return question.text unless question.kind == 'title'
case condition[:type]
when 'title-length-at-least'
"タイトルは #{ condition[:length] } 文字以上?"
when 'title-contains'
"題名に「#{ condition[:text] }」が含まれる?"
else
question.text
end
end
def example_answers_json question
question
.gekanator_question_examples
.group_by(&:post_id)
.transform_values { |examples| aggregate_answer(examples) }
end
def aggregate_answer examples
examples
.group_by(&:answer)
.map { |answer, grouped| [answer, grouped.sum(&:weight), grouped.max_by(&:updated_at)&.updated_at] }
.sort_by { |(_answer, weight, updated_at)| [-weight, -(updated_at&.to_f || 0)] }
.first
&.first
end
def deprecated_tag_keys_for questions
tag_keys = questions.filter_map { |question|
condition = condition_json(question.condition)
next unless condition['type'] == 'tag'
condition['key'].to_s.presence
}.uniq
return {} if tag_keys.empty?
categories = []
names = []
tag_keys.each do |key|
category, name = parse_tag_key(key)
categories << category
names << name
end
Tag
.joins(:tag_name)
.where(category: categories.uniq)
.where(tag_names: { name: names.uniq })
.where.not(deprecated_at: nil)
.pluck('tags.category', 'tag_names.name')
.each_with_object({ }) do |(category, name), h|
h["#{ category }:#{ name }"] = true
end
end
def hidden_question? condition, deprecated_tag_keys
condition[:type] == 'tag' && deprecated_tag_keys[condition[:key].to_s]
end
def parse_tag_key key
parts = key.to_s.split(':')
[parts.first.to_s, parts.drop(1).join(':')]
end
end
+49
ファイルの表示
@@ -0,0 +1,49 @@
class MaterialSyncSuppressionsController < ApplicationController
def index
return head :unauthorized unless current_user
return head :forbidden unless current_user.gte_member?
suppressions =
MaterialSyncSuppression
.includes(:created_by_user)
.order(created_at: :desc, id: :desc)
render json: { suppressions: suppressions.map { |s| repr(s) } }
end
def create
return head :unauthorized unless current_user
return head :forbidden unless current_user.gte_member?
suppression =
MaterialSyncSuppressionRegistrar.create!(suppression_params,
created_by_user: current_user)
render json: repr(suppression), status: :created
rescue ActiveRecord::RecordInvalid => e
render_validation_error e.record
end
private
def suppression_params
params.permit(:source_kind,
:source_uri,
:drive_path,
:drive_file_id,
:reason)
end
def repr suppression
{ id: suppression.id,
source_kind: suppression.source_kind,
source_uri: suppression.source_uri,
drive_path: suppression.drive_path,
drive_file_id: suppression.drive_file_id,
normalized_source_key: suppression.normalized_source_key,
reason: suppression.reason,
created_by_user: suppression.created_by_user&.as_json(UserRepr::BASE),
created_at: suppression.created_at,
updated_at: suppression.updated_at }
end
end
+54
ファイルの表示
@@ -0,0 +1,54 @@
class MaterialVersionsController < ApplicationController
def index
page = (params[:page].presence || 1).to_i
limit = (params[:limit].presence || 20).to_i
page = 1 if page < 1
limit = 1 if limit < 1
offset = (page - 1) * limit
q = MaterialVersion.includes(:created_by_user)
q = q.where(material_id: params[:material_id]) if params[:material_id].present?
q = q.where(tag_name: params[:tag].to_s.strip) if params[:tag].present?
q = q.where(event_type: params[:event_type]) if params[:event_type].present?
count = q.except(:order, :limit, :offset).count
versions =
q
.order(Arel.sql('material_versions.created_at DESC, material_versions.id DESC'))
.limit(limit)
.offset(offset)
render json: { versions: serialise_versions(versions), count: }
end
private
def serialise_versions rows
rows.map do |row|
{ id: row.id,
material_id: row.material_id,
version_no: row.version_no,
event_type: row.event_type,
tag_id: row.tag_id,
tag_name: row.tag_name,
tag_category: row.tag_category,
url: row.url,
source_kind: row.source_kind,
source_uri: row.source_uri,
source_path: row.source_path,
source_file_id: row.source_file_id,
normalized_source_key: row.normalized_source_key,
file_blob_id: row.file_blob_id,
file_filename: row.file_filename,
file_content_type: row.file_content_type,
file_byte_size: row.file_byte_size,
file_sha256: row.file_sha256,
export_paths_json: row.export_paths_hash,
discarded_at: row.discarded_at,
created_by_user: row.created_by_user&.as_json(UserRepr::BASE),
created_at: row.created_at }
end
end
end
+547
ファイルの表示
@@ -0,0 +1,547 @@
require 'set'
class MaterialsController < ApplicationController
rescue_from MaterialZipExporter::EmptyExportError, with: :render_zip_empty
rescue_from MaterialZipExporter::DuplicatePathError, with: :render_zip_duplicate_path
rescue_from MaterialZipExporter::MissingFileError, with: :render_zip_missing_file
def index
page = (params[:page].presence || 1).to_i
limit = (params[:limit].presence || 20).to_i
page = 1 if page < 1
limit = 1 if limit < 1
offset = (page - 1) * limit
filters = material_index_filters
tag_graph = material_index_tag_graph(filters)
q = Material.includes(:material_export_items,
thumbnail_attachment: :blob,
file_attachment: :blob,
tag: :tag_name)
q = q.where(tag_id: nil) if filters[:tag_state] == 'untagged'
q = q.where.not(tag_id: nil) if filters[:tag_state] == 'tagged'
q = apply_material_tag_filter(q, filters, tag_graph)
q = q.where('materials.created_at >= ?', filters[:created_from]) if filters[:created_from]
q = q.where('materials.created_at <= ?', filters[:created_to]) if filters[:created_to]
q = q.where('materials.updated_at >= ?', filters[:updated_from]) if filters[:updated_from]
q = q.where('materials.updated_at <= ?', filters[:updated_to]) if filters[:updated_to]
q = material_index_join_tag_name(q) if material_index_needs_tag_name?(filters)
q = material_index_join_file_blob(q) if material_index_needs_file_blob?(filters)
q = apply_material_query(q, filters[:q]) if filters[:q].present?
q = apply_material_media_kind(q, filters[:media_kind])
count = q.distinct.count(:id)
materials =
q
.order(Arel.sql(material_index_order_sql(filters)))
.limit(limit)
.offset(offset)
.to_a
response = { materials: MaterialRepr.list_many(materials, host: request.base_url),
count: }
if filters[:tag_id]
response[:tag_scope] = material_index_tag_scope(filters, tag_graph)
end
if filters[:group_by] == 'parent_tag'
response[:groups] = material_index_groups(materials, tag_graph)
end
render json: response
end
def show
material =
Material
.includes(:tag, :material_export_items)
.with_attached_thumbnail
.with_attached_file
.find_by(id: params[:id])
return head :not_found unless material
wiki_page_body = material.tag&.tag_name&.wiki_page&.current_revision&.body
render json: MaterialRepr.base(material, host: request.base_url).merge(wiki_page_body:)
end
def create
return head :unauthorized unless current_user
return head :forbidden unless current_user.gte_member?
tag_name_raw = params[:tag].to_s.strip
file = params[:file]
file_sha256 = MaterialFileSha256.from_upload(file)
url = params[:url].to_s.strip.presence
return render_unprocessable_entity('タグは必須です.', field: :tag) if tag_name_raw.blank?
if file.blank? && url.blank?
return render_validation_error fields: { file: ['ファイルまたは URL は必須です.'],
url: ['ファイルまたは URL は必須です.'] }
end
block = MaterialImportBlockMatcher.match_for_sha256(file_sha256)
return render_material_import_block(block) if block
uploaded_blob = build_uploaded_material_blob!(file, file_sha256)
material = nil
begin
Material.transaction do
tag = resolve_material_tag!(tag_name_raw)
material = Material.new(tag:, url:,
created_by_user: current_user,
updated_by_user: current_user)
material.file.attach(uploaded_blob) if uploaded_blob
material.save!
TagVersioning.record_tag_snapshot!(tag, created_by_user: current_user)
upsert_export_paths!(material)
MaterialVersionRecorder.record!(material:, event_type: :create,
created_by_user: current_user)
end
rescue StandardError
uploaded_blob&.purge_later
raise
end
if material
log_thumbnail_generation(material, MaterialThumbnailGenerator.generate!(material))
render json: MaterialRepr.base(material, host: request.base_url), status: :created
else
render_validation_error material
end
end
def update
return head :unauthorized unless current_user
return head :forbidden unless current_user.gte_member?
material = Material.with_attached_file.find_by(id: params[:id])
return head :not_found unless material
tag_name_raw = params[:tag].to_s.strip
file = params[:file]
file_sha256 = MaterialFileSha256.from_upload(file)
url = params.key?(:url) ? params[:url].to_s.strip.presence : material.url
return render_unprocessable_entity('タグは必須です.', field: :tag) if tag_name_raw.blank?
if file.blank? && url.blank? && !material.file.attached?
return render_validation_error fields: { file: ['ファイルまたは URL は必須です.'],
url: ['ファイルまたは URL は必須です.'] }
end
block = MaterialImportBlockMatcher.match_for_sha256(file_sha256)
return render_material_import_block(block) if block
uploaded_blob = build_uploaded_material_blob!(file, file_sha256)
begin
Material.transaction do
MaterialVersionRecorder.ensure_snapshot!(material, created_by_user: current_user)
tag = resolve_material_tag!(tag_name_raw)
material.assign_attributes(tag:, url:, updated_by_user: current_user)
if uploaded_blob
material.file.attach(uploaded_blob)
elsif params.key?(:url) && url.present? && file.blank?
material.file.detach
end
material.save!
TagVersioning.record_tag_snapshot!(tag, created_by_user: current_user)
upsert_export_paths!(material)
MaterialVersionRecorder.record!(material:, event_type: :update,
created_by_user: current_user)
end
rescue StandardError
uploaded_blob&.purge_later
raise
end
log_thumbnail_generation(material, MaterialThumbnailGenerator.generate!(material))
render json: MaterialRepr.base(material, host: request.base_url)
end
def destroy
return head :unauthorized unless current_user
return head :forbidden unless current_user.gte_member?
material = Material.find_by(id: params[:id])
return head :not_found unless material
Material.transaction do
MaterialVersionRecorder.ensure_snapshot!(material, created_by_user: current_user)
material.discard!
MaterialVersionRecorder.record!(material:, event_type: :discard,
created_by_user: current_user)
end
head :no_content
end
def download
zip = MaterialZipExporter.new(profile: params[:profile],
tag_id: params[:tag_id]).export
profile = params[:profile].presence || 'legacy_drive'
send_data zip,
type: 'application/zip',
disposition: 'attachment',
filename: "btrc-materials-#{ profile }.zip"
end
private
def material_index_filters
tag_state = params[:tag_state].to_s.presence
tag_state = 'untagged' if bool?(:unclassified)
tag_state = 'all' unless ['all', 'tagged', 'untagged'].include?(tag_state)
media_kind = params[:media_kind].to_s.presence
unless ['all', 'image', 'video', 'audio', 'file_other', 'url_only'].include?(media_kind)
media_kind = 'all'
end
sort = params[:sort].to_s.presence
unless ['created_at', 'updated_at', 'tag_name', 'media_kind', 'file_byte_size',
'version_no', 'id'].include?(sort)
sort = 'created_at'
end
direction = params[:direction].to_s.downcase
direction = 'desc' unless ['asc', 'desc'].include?(direction)
group_by = params[:group_by].to_s.presence
group_by = 'none' unless ['none', 'parent_tag'].include?(group_by)
tag_id = params[:tag_id].to_i
tag_id = nil if tag_id <= 0
{ q: params[:q].to_s.strip.presence,
tag_state:,
media_kind:,
tag_id:,
include_descendants: bool?(:include_descendants),
group_by:,
created_from: parse_time_param(:created_from),
created_to: parse_time_param(:created_to),
updated_from: parse_time_param(:updated_from),
updated_to: parse_time_param(:updated_to),
sort:,
direction: }
end
def parse_time_param name
value = params[name].to_s.strip
return nil if value.blank?
Time.zone.parse(value)
rescue ArgumentError
nil
end
def resolve_material_tag! tag_name_raw
tag_name = TagName.find_undiscard_or_create_by!(name: tag_name_raw)
tag = tag_name.tag
tag || Tag.create!(tag_name:, category: :material)
end
def material_index_needs_tag_name? filters
filters[:q].present? || filters[:sort] == 'tag_name'
end
def material_index_needs_file_blob? filters
filters[:q].present? ||
filters[:media_kind] != 'all' ||
['media_kind', 'file_byte_size'].include?(filters[:sort])
end
def material_index_join_tag_name q
q.left_joins(tag: :tag_name)
end
def material_index_join_file_blob q
q.joins(<<~SQL.squish)
LEFT JOIN active_storage_attachments material_file_attachments
ON material_file_attachments.record_type = 'Material'
AND material_file_attachments.record_id = materials.id
AND material_file_attachments.name = 'file'
LEFT JOIN active_storage_blobs material_file_blobs
ON material_file_blobs.id = material_file_attachments.blob_id
SQL
end
def apply_material_query q, term
like = "%#{ ActiveRecord::Base.sanitize_sql_like(term) }%"
q.where('tag_names.name LIKE :q OR materials.url LIKE :q OR ' \
'material_file_blobs.filename LIKE :q',
q: like)
end
def apply_material_media_kind q, media_kind
case media_kind
when 'image'
q.where('material_file_blobs.content_type LIKE ?', 'image/%')
when 'video'
q.where('material_file_blobs.content_type LIKE ?', 'video/%')
when 'audio'
q.where('material_file_blobs.content_type LIKE ?', 'audio/%')
when 'file_other'
q.where('material_file_attachments.id IS NOT NULL')
.where.not('material_file_blobs.content_type LIKE ?', 'image/%')
.where.not('material_file_blobs.content_type LIKE ?', 'video/%')
.where.not('material_file_blobs.content_type LIKE ?', 'audio/%')
when 'url_only'
q.where('material_file_attachments.id IS NULL').where.not(url: [nil, ''])
else
q
end
end
def apply_material_tag_filter q, filters, tag_graph
return q unless filters[:tag_id]
tag_ids =
if tag_graph
tag_graph[:scope_tag_ids]
else
[filters[:tag_id]]
end
q.where(tag_id: tag_ids)
end
def material_index_tag_graph filters
return nil unless filters[:tag_id]
children_by_parent_id = Hash.new { |h, k| h[k] = [] }
TagImplication.pluck(:parent_tag_id, :tag_id).each do |parent_id, child_id|
children_by_parent_id[parent_id] << child_id
end
scope_tag_ids =
if filters[:include_descendants]
collect_material_descendant_ids(filters[:tag_id], children_by_parent_id)
else
Set.new([filters[:tag_id]])
end
tags_by_id =
Tag
.joins(:tag_name)
.where(id: scope_tag_ids)
.pluck('tags.id', 'tag_names.name', 'tags.category', 'tags.deprecated_at')
.each_with_object({ }) do |(id, name, category, deprecated_at), hash|
hash[id] = { id:, name:, category:, deprecated: deprecated_at.present? }
end
selected_tag = tags_by_id[filters[:tag_id]]
return nil unless selected_tag
group_tag_id_by_tag_id = { }
children_by_parent_id[filters[:tag_id]].each do |child_tag_id|
assign_material_group_branch(child_tag_id,
children_by_parent_id:,
tags_by_id:,
group_tag_id_by_tag_id:,
current_group_tag_id: nil,
seen: Set.new([filters[:tag_id]]))
end
{ selected_tag_id: filters[:tag_id],
selected_tag:,
scope_tag_ids: scope_tag_ids.to_a,
tags_by_id:,
group_tag_id_by_tag_id: }
end
def collect_material_descendant_ids selected_tag_id, children_by_parent_id
ids = Set.new([selected_tag_id])
stack = [selected_tag_id]
until stack.empty?
tag_id = stack.pop
children_by_parent_id[tag_id].each do |child_tag_id|
next if ids.include?(child_tag_id)
ids << child_tag_id
stack << child_tag_id
end
end
ids
end
def assign_material_group_branch tag_id, children_by_parent_id:, tags_by_id:,
group_tag_id_by_tag_id:, current_group_tag_id:, seen:
return if seen.include?(tag_id)
tag = tags_by_id[tag_id]
return unless tag
seen = seen.dup << tag_id
next_group_tag_id = current_group_tag_id
next_group_tag_id = tag_id if next_group_tag_id.nil? && !tag[:deprecated]
group_tag_id_by_tag_id[tag_id] = next_group_tag_id if next_group_tag_id
children_by_parent_id[tag_id].each do |child_tag_id|
assign_material_group_branch(child_tag_id,
children_by_parent_id:,
tags_by_id:,
group_tag_id_by_tag_id:,
current_group_tag_id: next_group_tag_id,
seen:)
end
end
def material_index_groups materials, tag_graph
return [] unless tag_graph
groups_by_key = { }
materials.each do |material|
next unless material.tag_id
group_tag_id =
if material.tag_id == tag_graph[:selected_tag_id]
tag_graph[:selected_tag_id]
else
tag_graph[:group_tag_id_by_tag_id][material.tag_id] || tag_graph[:selected_tag_id]
end
group_tag = tag_graph[:tags_by_id][group_tag_id]
next unless group_tag
key = "tag:#{ group_tag_id }"
group =
groups_by_key[key] ||= { key:,
tag: group_tag_repr(group_tag),
material_ids: [],
count: 0 }
group[:material_ids] << material.id
group[:count] += 1
end
groups_by_key.values.sort_by { |group| group[:tag][:name] }
end
def material_index_tag_scope filters, tag_graph
return nil unless tag_graph
{
tag: group_tag_repr(tag_graph[:selected_tag]),
include_descendants: filters[:include_descendants]
}
end
def group_tag_repr group_tag
{ id: group_tag[:id],
name: group_tag[:name],
category: group_tag[:category] }
end
def material_index_order_sql filters
direction = filters[:direction] == 'asc' ? 'ASC' : 'DESC'
sort_sql =
case filters[:sort]
when 'tag_name'
'tag_names.name'
when 'media_kind'
material_media_kind_sql
when 'file_byte_size'
'material_file_blobs.byte_size'
when 'updated_at'
'materials.updated_at'
when 'version_no'
'materials.version_no'
when 'id'
'materials.id'
else
'materials.created_at'
end
"#{ sort_sql } #{ direction }, materials.id #{ direction }"
end
def material_media_kind_sql
"CASE " \
"WHEN material_file_attachments.id IS NULL AND materials.url IS NOT NULL THEN 5 " \
"WHEN material_file_blobs.content_type LIKE 'image/%' THEN 1 " \
"WHEN material_file_blobs.content_type LIKE 'video/%' THEN 2 " \
"WHEN material_file_blobs.content_type LIKE 'audio/%' THEN 3 " \
"WHEN material_file_attachments.id IS NOT NULL THEN 4 " \
"ELSE 6 END"
end
def upsert_export_paths! material
raw = params[:export_paths]
return if raw.blank?
export_paths = raw.respond_to?(:to_unsafe_h) ? raw.to_unsafe_h : raw.to_h
export_paths.each do |profile, export_path|
profile = profile.to_s
export_path = export_path.to_s.strip
item = material.material_export_items.find_or_initialize_by(profile:)
if export_path.blank?
item.destroy! if item.persisted?
next
end
item.export_path = export_path
item.enabled = true
item.created_by_user ||= current_user
item.save!
end
end
def render_zip_empty
render_unprocessable_entity('ZIP export 対象の素材がありません.')
end
def render_zip_duplicate_path error
render_unprocessable_entity("ZIP export path が重複してゐます: #{ error.message }")
end
def render_zip_missing_file error
missing_files = error.missing_files.map do |missing_file|
{ material_id: missing_file.material_id,
export_path: missing_file.export_path,
blob_id: missing_file.blob_id,
filename: missing_file.filename }
end
render json: { type: 'validation_error',
message: 'ZIP export に必要な素材ファイルが欠損しています.',
errors: { },
base_errors: ['ZIP export に必要な素材ファイルが欠損しています.'],
missing_files: },
status: :unprocessable_entity
end
def build_uploaded_material_blob! file, file_sha256
return nil unless file
file.tempfile.rewind
blob = ActiveStorage::Blob.create_and_upload!(
io: file.tempfile,
filename: file.original_filename,
content_type: file.content_type,
)
MaterialFileSha256.assign_metadata_sha256!(blob, file_sha256)
blob
ensure
file.tempfile.rewind if file&.tempfile
end
def render_material_import_block block
render_validation_error fields: { file: ["抑止された素材です: #{ block.reason }"] }
end
def log_thumbnail_generation material, result
material.reload
Rails.logger.info(
"Material thumbnail generation: material_id=#{ material.id } " \
"result=#{ result } " \
"thumbnail_attached=#{ material.thumbnail.attached? } " \
"content_type=#{ material.content_type }")
end
end
+91 -21
ファイルの表示
@@ -1,26 +1,69 @@
class NicoTagsController < ApplicationController
def index
limit = (params[:limit] || 20).to_i
cursor = params[:cursor].presence
name = params[:name].presence
linked_tag = params[:linked_tag].presence
link_status = params[:link_status].presence
order = params[:order].to_s.split(':', 2).map(&:strip)
order[0] = 'updated_at' unless order[0].in?(['name', 'created_at', 'updated_at'])
unless order[1].in?(['asc', 'desc'])
order[1] = order[0] == 'name' ? 'asc' : 'desc'
end
page = (params[:page].presence || 1).to_i
limit = (params[:limit].presence || 20).to_i
page = 1 if page < 1
limit = 1 if limit < 1
post_tag_max_sql =
PostTag
.select('tag_id, MAX(created_at) AS max_created_at')
.group('tag_id')
.to_sql
q = Tag.nico_tags
.joins(:tag_name)
.joins("LEFT JOIN (#{ post_tag_max_sql }) post_tag_max " \
'ON post_tag_max.tag_id = tags.id')
.includes(:tag_name, tag_name: :wiki_page, linked_tags: { tag_name: :wiki_page })
.order(updated_at: :desc)
q = q.where('tags.updated_at < ?', Time.iso8601(cursor)) if cursor
tags = q.limit(limit + 1).to_a
next_cursor = nil
if tags.size > limit
next_cursor = tags.last.updated_at.iso8601(6)
tags = tags.first(limit)
q = q.where('tag_names.name LIKE ?', "%#{ name }%") if name
if linked_tag
linked_tag_ids =
Tag
.joins(:tag_name)
.where('tag_names.name LIKE ?', "%#{ linked_tag }%")
.pluck(:id)
linked_nico_tag_ids = NicoTagRelation.where(tag_id: linked_tag_ids).pluck(:nico_tag_id)
q = q.where(id: linked_nico_tag_ids)
end
if link_status.in?(['linked', 'unlinked'])
exists_sql =
'EXISTS (SELECT 1 FROM nico_tag_relations ' \
'WHERE nico_tag_relations.nico_tag_id = tags.id)'
q = link_status == 'linked' ? q.where(exists_sql) : q.where("NOT #{ exists_sql }")
end
count = q.count
sort_sql =
case order[0]
when 'name'
'tag_names.name'
when 'updated_at'
'post_tag_max.max_created_at'
else
"tags.#{ order[0] }"
end
tags = q.reselect('tags.*',
Arel.sql('post_tag_max.max_created_at AS recent_post_tag_created_at'))
.order(Arel.sql("#{ sort_sql } #{ order[1] }, tags.id #{ order[1] }"))
.limit(limit)
.offset((page - 1) * limit)
.to_a
render json: { tags: tags.map { |tag|
TagRepr.base(tag).merge(linked_tags: tag.linked_tags.map { |lt|
TagRepr.base(lt)
})
}, next_cursor: }
TagRepr.base(tag).merge(
recent_post_tag_created_at: tag.recent_post_tag_created_at,
linked_tags: tag.linked_tags.map { |lt| TagRepr.base(lt) })
}, count: }
end
def update
@@ -30,15 +73,42 @@ class NicoTagsController < ApplicationController
id = params[:id].to_i
tag = Tag.find(id)
return head :bad_request if tag.category != 'nico'
return render_bad_request('ニコニコ・タグを指定してください.') unless tag.nico?
linked_tag_names = params[:tags].to_s.split(' ')
linked_tags = Tag.normalise_tags(linked_tag_names, with_tagme: false)
return head :bad_request if linked_tags.any? { |t| t.category == 'nico' }
linked_tag_names = params[:tags].to_s.split
linked_tags = nil
tag.linked_tags = linked_tags
tag.save!
ApplicationRecord.transaction do
linked_tags = Tag.normalise_tags!(linked_tag_names, with_tagme: false,
with_no_deerjikist: false)
if linked_tags.any? { |t| t.nico? }
raise Tag::NicoTagNormalisationError
end
TagVersioning.record_tag_snapshots!(linked_tags, created_by_user: current_user)
tag.linked_tags = linked_tags
tag.save!
NicoTagVersionRecorder.record!(tag:, event_type: :update, created_by_user: current_user)
end
render json: tag.linked_tags.map { |t| TagRepr.base(t) }, status: :ok
rescue Tag::NicoTagNormalisationError
render_validation_error fields: { tags: ['ニコニコ・タグ同士は連携できません.'] }
rescue ActiveRecord::RecordInvalid => e
render_nico_tag_form_record_invalid e.record
end
private
def render_nico_tag_form_record_invalid record
if record.is_a?(TagName) || record.is_a?(Tag)
render_validation_error fields: { tags: record.errors.full_messages.map { |message|
"タグ名 “#{ record.name }”: #{ message }"
} }
else
render_validation_error record
end
end
end
+124
ファイルの表示
@@ -0,0 +1,124 @@
class PostVersionsController < ApplicationController
def index
post_id = params[:post].presence
tag_id = params[:tag].presence
page = (params[:page].presence || 1).to_i
limit = (params[:limit].presence || 20).to_i
page = 1 if page < 1
limit = 1 if limit < 1
offset = (page - 1) * limit
tag_name =
if tag_id
TagName.joins(:tag).find_by(tag: { id: tag_id })
end
return render json: { versions: [], count: 0 } if tag_id && tag_name.blank?
q = PostVersion.joins(<<~SQL.squish)
LEFT JOIN
post_versions prev
ON
prev.post_id = post_versions.post_id
AND prev.version_no = post_versions.version_no - 1
SQL
.select('post_versions.*', 'prev.title AS prev_title', 'prev.url AS prev_url',
'prev.thumbnail_base AS prev_thumbnail_base', 'prev.tags AS prev_tags',
'prev.video_ms AS prev_video_ms',
'prev.original_created_from AS prev_original_created_from',
'prev.original_created_before AS prev_original_created_before')
q = q.where('post_versions.post_id = ?', post_id) if post_id
if tag_name
escaped = ActiveRecord::Base.sanitize_sql_like(tag_name.name)
q = q.where(("CONCAT(' ', post_versions.tags, ' ') LIKE :kw " +
"OR CONCAT(' ', prev.tags, ' ') LIKE :kw"),
kw: "% #{ escaped } %")
end
count = q.except(:select, :order, :limit, :offset).count
versions = q.order(Arel.sql('post_versions.created_at DESC, post_versions.id DESC'))
.limit(limit)
.offset(offset)
render json: { versions: serialise_versions(versions), count: }
end
private
def serialise_versions rows
user_ids = rows.map(&:created_by_user_id).compact.uniq
users_by_id = User.where(id: user_ids).pluck(:id, :name).to_h
rows.map do |row|
cur_tags = split_tags(row.tags)
prev_tags = split_tags(row.attributes['prev_tags'])
{
post_id: row.post_id,
version_no: row.version_no,
event_type: row.event_type,
title: {
current: row.title,
prev: row.attributes['prev_title']
},
url: {
current: row.url,
prev: row.attributes['prev_url']
},
thumbnail: {
current: nil,
prev: nil
},
thumbnail_base: {
current: row.thumbnail_base,
prev: row.attributes['prev_thumbnail_base']
},
video_ms: {
current: row.video_ms,
prev: row.attributes['prev_video_ms']
},
tags: build_version_tags(cur_tags, prev_tags),
original_created_from: {
current: row.original_created_from&.iso8601,
prev: row.attributes['prev_original_created_from']&.iso8601
},
original_created_before: {
current: row.original_created_before&.iso8601,
prev: row.attributes['prev_original_created_before']&.iso8601
},
created_at: row.created_at.iso8601,
created_by_user:
if row.created_by_user_id
{
id: row.created_by_user_id,
name: users_by_id[row.created_by_user_id]
}
end
}
end
end
def build_version_tags(cur_tags, prev_tags)
(cur_tags | prev_tags).map do |name|
type =
if cur_tags.include?(name) && prev_tags.include?(name)
'context'
elsif cur_tags.include?(name)
'added'
else
'removed'
end
{
name:,
type:
}
end
end
def split_tags(tags)
tags.to_s.split(/\s+/).reject(&:blank?)
end
end
+667 -47
ファイルの表示
@@ -1,5 +1,10 @@
class PostsController < ApplicationController
Event = Struct.new(:post, :tag, :user, :change_type, :timestamp, keyword_init: true)
MAX_BULK_REQUEST_BYTES = 40 * 1024 * 1024
class VideoMsParseError < ArgumentError
;
end
def index
url = params[:url].presence
@@ -44,7 +49,10 @@ class PostsController < ApplicationController
filtered_posts
.joins("LEFT JOIN (#{ pt_max_sql }) pt_max ON pt_max.post_id = posts.id")
.reselect('posts.*', Arel.sql("#{ updated_at_all_sql } AS updated_at_all"))
.preload(tags: { tag_name: :wiki_page })
.preload(:uploaded_user, :parents, :children,
active_post_tags: [:sections,
{ tag: [:deerjikists, :materials,
{ tag_name: :wiki_page }] }])
.with_attached_thumbnail
q = q.where('posts.url LIKE ?', "%#{ url }%") if url
@@ -95,7 +103,11 @@ class PostsController < ApplicationController
end
def random
post = filtered_posts.preload(tags: { tag_name: :wiki_page })
post = filtered_posts.preload(:uploaded_user, :parents, :children,
active_post_tags: [:sections,
{ tag: [:deerjikists, :materials,
{ tag_name: :wiki_page }] }])
.with_attached_thumbnail
.order('RAND()')
.first
return head :not_found unless post
@@ -103,43 +115,170 @@ class PostsController < ApplicationController
render json: PostRepr.base(post, current_user)
end
def metadata
return head :unauthorized unless current_user
return head :forbidden unless current_user.gte_member?
return render_bad_request('URL は必須です.') if params[:url].blank?
normal_url = PostUrlNormaliser.normalise(params[:url].to_s)
return render_validation_error(fields: { url: ['URL が不正です.'] }) if normal_url.blank?
Preview::UrlSafety.validate(normal_url)
existing_post = Post.with_attached_thumbnail.find_by(url: normal_url)
if existing_post.present?
return render json: {
url: normal_url,
title: nil,
thumbnail_base: nil,
tags: nil,
display_tags: [],
original_created_from: nil,
original_created_before: nil,
duration: nil,
video_ms: nil,
field_warnings: { },
base_warnings: [],
existing_post_id: existing_post.id,
existing_post: compact_post(existing_post.id) }
end
metadata = PostMetadataFetcher.fetch(normal_url)
field_warnings = { }
field_warnings[:title] = ['タイトルを取得できませんでした.'] if metadata[:title].blank?
if metadata[:thumbnail_base].blank?
field_warnings[:thumbnail_base] = ['サムネールを取得できませんでした.']
end
render json: {
url: normal_url,
title: metadata[:title],
thumbnail_base: metadata[:thumbnail_base],
tags: metadata[:tags],
display_tags: metadata[:display_tags],
original_created_from: metadata[:original_created_from],
original_created_before: metadata[:original_created_before],
duration: metadata[:duration],
video_ms: metadata[:video_ms],
field_warnings: field_warnings,
base_warnings: [],
existing_post_id: nil,
existing_post: nil }
rescue ArgumentError => e
render_bad_request e.message
rescue Preview::UrlSafety::UnsafeUrl => e
render_validation_error fields: { url: [e.message] }
rescue Preview::HttpFetcher::FetchFailed,
Preview::HttpFetcher::FetchTimeout,
Preview::HttpFetcher::ResponseTooLarge
render json: {
url: normal_url,
title: nil,
thumbnail_base: nil,
tags: nil,
display_tags: [],
original_created_from: nil,
original_created_before: nil,
duration: nil,
video_ms: nil,
field_warnings: { url: ['自動取得に失敗しました.'] },
base_warnings: [],
existing_post_id: nil,
existing_post: nil }
end
def show
post = Post.includes(tags: { tag_name: :wiki_page }).find_by(id: params[:id])
post =
Post
.includes(:uploaded_user, :parents, :children,
active_post_tags: [:sections,
{ tag: [:deerjikists, :materials,
{ tag_name: :wiki_page }] }])
.with_attached_thumbnail
.find_by(id: params[:id])
return head :not_found unless post
render json: PostRepr.base(post, current_user)
.merge(tags: build_tag_tree_for(post.tags),
related: post.related(limit: 20))
parent_posts = post.parents.with_attached_thumbnail.order(:id).to_a
child_posts = post.children.with_attached_thumbnail.order(:id).to_a
sibling_posts = sibling_posts_by_parent(parent_posts.map(&:id))
related = post.related(limit: 20).to_a
render json: PostRepr.detail(post, current_user,
parent_posts:,
child_posts:,
sibling_posts:,
related:)
.merge(tags: build_tag_tree_for(post))
end
def create
return head :unauthorized unless current_user
return head :forbidden unless current_user.gte_member?
# TODO: サイトに応じて thumbnail_base 設定
title = params[:title].presence
url = params[:url]
thumbnail = params[:thumbnail]
tag_names = params[:tags].to_s.split
original_created_from = params[:original_created_from]
original_created_before = params[:original_created_before]
post = Post.new(title:, url:, thumbnail_base: nil, uploaded_user: current_user,
original_created_from:, original_created_before:)
post.thumbnail.attach(thumbnail)
if post.save
post.resized_thumbnail!
tags = Tag.normalise_tags(tag_names)
tags = Tag.expand_parent_tags(tags)
sync_post_tags!(post, tags)
post.reload
render json: PostRepr.base(post), status: :created
else
render json: { errors: post.errors.full_messages }, status: :unprocessable_entity
preflight = PostCreatePreflight.new(
attributes: post_create_attributes,
thumbnail: params[:thumbnail]).run
return render json: dry_run_json(preflight) if bool?(:dry)
if preflight[:existing_post_id].present?
post = Post.new(url: preflight[:url])
post.errors.add :url, :taken
return render_post_form_record_invalid post
end
post = PostCreator.new(actor: current_user,
attributes: post_create_attributes.merge(
preflight.slice(
:url,
:title,
:thumbnail_base,
:tags,
:parent_post_ids,
:original_created_from,
:original_created_before,
:duration,
:video_ms,
:direct_tag_specs,
:default_tag_specs,
:snapshot_tag_specs,
:post_tag_specs,
:tag_sections,
:normalised_parent_post_ids).symbolize_keys).merge(
thumbnail: params[:thumbnail])).create!
post.reload
render json: PostRepr.base(post), status: :created
rescue PostCreatePreflight::ValidationFailed => e
render_validation_error fields: e.fields, base: e.base_errors
rescue Tag::NicoTagNormalisationError
head :bad_request
render_validation_error fields: { tags: 'ニコニコ・タグは直接指定できません.' }
rescue Tag::DeprecatedTagNormalisationError
render_unprocessable_entity '廃止済みタグは付与できません.', field: :tags
rescue Tag::SectionLiteralParseError
render_validation_error fields: { tags: ['タグ区間の記法が不正です.'] }
rescue PostCreator::VideoMsParseError
render_validation_error fields: { video_ms: ['動画時間の記法が不正です.'] }
rescue Post::RemoteThumbnailFetchFailed
render_validation_error fields: { thumbnail_base: ['サムネイル画像の取得に失敗しました.'] }
rescue MiniMagick::Error
render_validation_error fields: { thumbnail: ['サムネイル画像の変換に失敗しました.'] }
rescue ArgumentError => e
render_validation_error fields: { parent_post_ids: [e.message] }
rescue ActiveRecord::RecordInvalid => e
render_post_form_record_invalid e.record
end
def bulk
return head :unauthorized unless current_user
return head :forbidden unless current_user.gte_member?
return head :unsupported_media_type unless request.content_mime_type == Mime[:multipart_form]
return head :payload_too_large if request.content_length.to_i > MAX_BULK_REQUEST_BYTES
posts = parse_bulk_posts_manifest
thumbnails = parse_bulk_thumbnails(posts.length)
result = PostBulkCreator.new(actor: current_user, posts:, thumbnails:).run
render json: result
rescue JSON::ParserError
render_bad_request 'posts manifest の JSON が不正です.'
rescue ArgumentError => e
render_validation_error base: [e.message]
end
def viewed
@@ -160,27 +299,84 @@ class PostsController < ApplicationController
return head :unauthorized unless current_user
return head :forbidden unless current_user.gte_member?
force = bool?(:force)
merge = bool?(:merge)
return render_bad_request('force と merge は同時に指定できません.') if force && merge
base_version_no = parse_base_version_no
return render_bad_request('base_version_no は必須です.') if !(force) && !(base_version_no)
title = params[:title].presence
tag_names = params[:tags].to_s.split
original_created_from = params[:original_created_from]
original_created_before = params[:original_created_before]
parent_post_ids = parse_parent_post_ids
post = Post.find(params[:id].to_i)
if post.update(title:, original_created_from:, original_created_before:)
tags = post.tags.where(category: 'nico').to_a +
Tag.normalise_tags(tag_names, with_tagme: false)
tags = Tag.expand_parent_tags(tags)
sync_post_tags!(post, tags)
post = nil
conflict_json = nil
post.reload
json = post.as_json
json['tags'] = build_tag_tree_for(post.tags)
render json:, status: :ok
else
render json: post.errors, status: :unprocessable_entity
ApplicationRecord.transaction do
post = Post.lock.find(params[:id].to_i)
base_version = nil
base_snapshot = nil
current_snapshot = nil
unless force
base_version = post.post_versions.find_by!(version_no: base_version_no)
base_snapshot = post_snapshot_from_version(base_version)
current_snapshot = post_snapshot_from_record(post)
end
incoming_snapshot = post_incoming_snapshot(title:,
original_created_from:,
original_created_before:,
tag_names:,
video_ms_param: params[:video_ms],
duration_param: params[:duration],
parent_post_ids:)
snapshot_to_apply =
if force || post.version_no == base_version_no || current_snapshot == base_snapshot
incoming_snapshot
else
changes = post_snapshot_changes(base_snapshot, current_snapshot, incoming_snapshot)
conflicts = changes.select { |change| change[:conflict] }
if merge && conflicts.empty?
merge_post_snapshots(base_snapshot, current_snapshot, incoming_snapshot)
else
conflict_json = post_conflict_json(post:,
base_version_no:,
base_snapshot:,
current_snapshot:,
incoming_snapshot:,
changes:,
conflicts:)
raise ActiveRecord::Rollback
end
end
apply_post_snapshot!(post, snapshot_to_apply)
end
return render json: conflict_json, status: :conflict if conflict_json
post.reload
json = PostRepr.base(post, current_user)
json['tags'] = build_tag_tree_for(post)
render json:, status: :ok
rescue Tag::NicoTagNormalisationError
head :bad_request
render_validation_error fields: { tags: ['ニコニコ・タグは直接指定できません.'] }
rescue Tag::DeprecatedTagNormalisationError
render_unprocessable_entity '廃止済みタグは付与できません.', field: :tags
rescue Tag::SectionLiteralParseError
render_validation_error fields: { tags: ['タグ区間の記法が不正です.'] }
rescue VideoMsParseError
render_validation_error fields: { video_ms: ['動画時間の記法が不正です.'] }
rescue ArgumentError => e
render_validation_error fields: { parent_post_ids: [e.message] }
rescue ActiveRecord::RecordInvalid => e
render_post_form_record_invalid e.record
end
def changes
@@ -198,7 +394,7 @@ class PostsController < ApplicationController
pts = pts.where(post_id: id) if id.present?
pts = pts.where(tag_id:) if tag_id.present?
pts = pts.includes(:post, :created_user, :deleted_user,
tag: { tag_name: :wiki_page })
tag: [:deerjikists, :materials, { tag_name: :wiki_page }])
events = []
pts.each do |pt|
@@ -275,7 +471,7 @@ class PostsController < ApplicationController
def tagged_post_ids_for(name) =
Post.joins(tags: :tag_name).where(tag_names: { name: }).select(:id)
def sync_post_tags! post, desired_tags
def sync_post_tags! post, desired_tags, sections
desired_tags.each do |t|
t.save! if t.new_record?
end
@@ -294,13 +490,21 @@ class PostsController < ApplicationController
end
end
PostTagSection.where(post_id: post.id).destroy_all
sections.each do |tag_id, ranges|
ranges.each do |begin_ms, end_ms|
PostTagSection.create!(post_id: post.id, tag_id:, begin_ms:, end_ms:)
end
end
PostTag.where(post_id: post.id, tag_id: to_remove.to_a).kept.find_each do |pt|
pt.discard_by!(current_user)
end
end
def build_tag_tree_for tags
tags = tags.to_a
def build_tag_tree_for post
post_tags = post.active_post_tags.reject { |post_tag| post_tag.tag.deprecated? }
tags = post_tags.map(&:tag)
tag_ids = tags.map(&:id)
implications = TagImplication.where(parent_tag_id: tag_ids, tag_id: tag_ids)
@@ -315,6 +519,9 @@ class PostsController < ApplicationController
root_ids = tag_ids - child_ids
tags_by_id = tags.index_by(&:id)
sections_by_tag_id = post_tags.to_h { |post_tag|
[post_tag.tag_id, post_tag.sections.as_json(only: [:begin_ms, :end_ms])]
}
memo = { }
@@ -322,8 +529,10 @@ class PostsController < ApplicationController
tag = tags_by_id[tag_id]
return nil unless tag
sections = sections_by_tag_id.fetch(tag_id, [])
if path.include?(tag_id)
return TagRepr.base(tag).merge(children: [])
return TagRepr.inline(tag).merge(children: [], sections:)
end
if memo.key?(tag_id)
@@ -335,9 +544,420 @@ class PostsController < ApplicationController
children = child_ids.filter_map { |cid| build_node.(cid, new_path) }
memo[tag_id] = TagRepr.base(tag).merge(children:)
memo[tag_id] = TagRepr.inline(tag).merge(children:, sections:)
end
root_ids.filter_map { |id| build_node.call(id, []) }
end
def sibling_posts_by_parent parent_post_ids
return { } if parent_post_ids.blank?
implications =
PostImplication
.where(parent_post_id: parent_post_ids)
.includes(post: { thumbnail_attachment: :blob })
.order(:parent_post_id, :post_id)
implications.group_by(&:parent_post_id).transform_values { |items|
items.map(&:post)
}
end
def parse_parent_post_ids
raise ArgumentError, 'parent_post_ids は必須です.' unless params.key?(:parent_post_ids)
params[:parent_post_ids].to_s.split.map { |token|
id = Integer(token, exception: false)
raise ArgumentError, "親投稿 Id. が不正です: #{ token }" if id.nil? || id <= 0
id
}.uniq
end
def post_create_attributes
{ title: params[:title],
url: params[:url],
thumbnail_base: params[:thumbnail_base],
tags: params[:tags],
original_created_from: params[:original_created_from],
original_created_before: params[:original_created_before],
parent_post_ids: parse_parent_post_ids,
video_ms: params[:video_ms],
duration: params[:duration] }
end
def parse_bulk_posts_manifest
manifest = params[:posts]
raise ArgumentError, 'posts は必須です.' if manifest.blank?
raise ArgumentError, 'posts は JSON 文字列で指定してください.' unless manifest.is_a?(String)
posts = JSON.parse(manifest)
raise ArgumentError, 'posts は配列で指定してください.' unless posts.is_a?(Array)
raise ArgumentError, '投稿件数は 1 件以上必要です.' if posts.empty?
raise ArgumentError, '投稿件数が多すぎます.' if posts.length > 100
raise ArgumentError, 'posts 要素の形式が不正です.' unless posts.all? { _1.is_a?(Hash) }
posts
end
def parse_bulk_thumbnails post_count
thumbnails = { }
raw = params[:thumbnails]
return thumbnails if raw.blank?
raise ArgumentError, 'thumbnail key が不正です.' unless raw.respond_to?(:to_unsafe_h)
raw.to_unsafe_h.each do |key, value|
raise ArgumentError, 'thumbnail key が不正です.' unless key.to_s.match?(/\A\d+\z/)
index = Integer(key, 10)
raise ArgumentError, 'thumbnail index が範囲外です.' if index.negative? || index >= post_count
raise ArgumentError, 'thumbnail index が重複しています.' if thumbnails.key?(index)
unless value.is_a?(ActionDispatch::Http::UploadedFile)
raise ArgumentError, 'thumbnail upload が不正です.'
end
thumbnails[index] = value
end
thumbnails
end
def compact_post post_id
return nil if post_id.blank?
post = Post.with_attached_thumbnail.find_by(id: post_id)
PostCompactRepr.base(post)
end
def dry_run_json preflight
preflight.slice(
:url,
:title,
:thumbnail_base,
:tags,
:display_tags,
:parent_post_ids,
:original_created_from,
:original_created_before,
:duration,
:video_ms,
:field_warnings,
:base_warnings,
:existing_post_id,
:existing_post)
end
def sync_parent_posts! post, parent_post_ids
if parent_post_ids.include?(post.id)
post.errors.add :parent_post_ids, '自分自身を親投稿にはできません.'
raise ActiveRecord::RecordInvalid, post
end
existing_ids = Post.where(id: parent_post_ids).pluck(:id)
missing_ids = parent_post_ids - existing_ids
if missing_ids.present?
post.errors.add :parent_post_ids,
"存在しない親投稿 Id. があります: #{ missing_ids.join(' ') }"
raise ActiveRecord::RecordInvalid, post
end
current_ids = post.parent_posts.pluck(:id)
ids_to_add = parent_post_ids - current_ids
ids_to_remove = current_ids - parent_post_ids
PostImplication.where(post_id: post.id, parent_post_id: ids_to_remove).delete_all
ids_to_add.each do |parent_post_id|
PostImplication.create_or_find_by!(post_id: post.id, parent_post_id:)
end
end
def parse_base_version_no
version_no = Integer(params[:base_version_no], exception: false)
if version_no&.positive?
version_no
else
nil
end
end
def post_snapshot_from_version version
{ title: version.title,
video_ms: version.respond_to?(:video_ms) ? version.video_ms : nil,
original_created_from: snapshot_time(version.original_created_from),
original_created_before: snapshot_time(version.original_created_before),
tag_names: editable_tag_names_from_version(version),
parent_post_ids: snapshot_parent_post_ids_from_version(version) }
end
def editable_tag_names_from_version version
version.tags.to_s.split.reject { |name| name.downcase.start_with?('nico:') }.sort
end
def post_snapshot_from_record post
{ title: post.title,
video_ms: post.video_ms,
original_created_from: snapshot_time(post.original_created_from),
original_created_before: snapshot_time(post.original_created_before),
tag_names: editable_tag_names_from_post(post),
parent_post_ids: post.parent_posts.order(:id).pluck(:id) }
end
def editable_tag_names_from_post post
post
.post_tags
.kept
.joins(tag: :tag_name)
.merge(Tag.not_nico)
.merge(Tag.where(deprecated_at: nil))
.includes(:sections, tag: :tag_name)
.order('tag_names.name')
.map do |post_tag|
name = post_tag.tag.tag_name.name
sections = post_tag.sections.sort_by(&:begin_ms)
next name if sections.empty?
"#{ name }#{ sections.map { Post.section_literal(_1) }.join }"
end
end
def post_incoming_snapshot title:, original_created_from:, original_created_before:,
tag_names:, video_ms_param:, duration_param:, parent_post_ids:
Tag.normalise_tags!(tag_names, with_tagme: false, deny_deprecated: true,
with_sections: true) =>
{ tags:, sections: }
tags = Tag.expand_parent_tags(tags).reject(&:deprecated?)
video_ms = normalise_video_ms(tags, video_ms_param:, duration_param:)
validate_video_sections!(video_ms, sections)
{ title:,
video_ms:,
original_created_from: snapshot_time(original_created_from),
original_created_before: snapshot_time(original_created_before),
tag_names: tags.uniq(&:id).map { |tag|
"#{ tag.name }#{ sections[tag.id].to_a.map { section_literal(_1) }.join }"
}.sort,
parent_post_ids: parent_post_ids.sort }
end
def snapshot_parent_post_ids_from_version version
if version.respond_to?(:parent_post_ids)
version.parent_post_ids.to_s.split.map { |id| id.to_i }.sort
elsif version.respond_to?(:parent_id) && version.parent_id
[version.parent_id]
else
[]
end
end
def snapshot_time value
return nil if value.blank?
value = Time.zone.parse(value.to_s) if value in String
value&.in_time_zone&.iso8601(6)
rescue ArgumentError, TypeError
value.to_s
end
def section_literal section
"[#{ Post.ms_to_time(section[0]) }-#{ section[1] ? Post.ms_to_time(section[1]) : '' }]"
end
def post_conflict_json post:, base_version_no:, base_snapshot:,
current_snapshot:, incoming_snapshot:, changes:, conflicts:
{ error: 'conflict',
message: '競合が発生しました.',
post_id: post.id,
base_version_no:,
current_version_no: post.version_no,
base: base_snapshot,
current: current_snapshot,
mine: incoming_snapshot,
changes:,
conflicts:,
mergeable: conflicts.empty? }
end
def post_snapshot_changes base_snapshot, current_snapshot, incoming_snapshot
[scalar_snapshot_change(:title, 'タイトル',
base_snapshot, current_snapshot, incoming_snapshot),
scalar_snapshot_change(:video_ms, '動画時間',
base_snapshot, current_snapshot, incoming_snapshot),
scalar_snapshot_change(:original_created_from, 'オリジナルの作成日時(以降)',
base_snapshot, current_snapshot, incoming_snapshot),
scalar_snapshot_change(:original_created_before, 'オリジナルの作成日時(より前)',
base_snapshot, current_snapshot, incoming_snapshot),
set_snapshot_change(:tag_names, 'タグ',
base_snapshot, current_snapshot, incoming_snapshot),
set_snapshot_change(:parent_post_ids, '親投稿',
base_snapshot, current_snapshot, incoming_snapshot)].compact
end
def scalar_snapshot_change field, label, base_snapshot, current_snapshot, incoming_snapshot
base = base_snapshot[field]
current = current_snapshot[field]
mine = incoming_snapshot[field]
return nil if current == base && mine == base
{ field:, label:, base:, current:, mine:,
changed_by_current: current != base,
changed_by_me: mine != base,
conflict: scalar_snapshot_conflict?(base, current, mine) }
end
def scalar_snapshot_conflict? base, current, mine
current != base && mine != base && current != mine
end
def set_snapshot_change field, label, base_snapshot, current_snapshot, incoming_snapshot
base = base_snapshot[field].to_a
current = current_snapshot[field].to_a
mine = incoming_snapshot[field].to_a
added_by_current = current - base
removed_by_current = base - current
added_by_me = mine - base
removed_by_me = base - mine
if (added_by_current.empty? &&
removed_by_current.empty? &&
added_by_me.empty? &&
removed_by_me.empty?)
return nil
end
{ field:, label:, base:, current:, mine:, added_by_current:, removed_by_current:,
added_by_me:, removed_by_me:,
changed_by_current: added_by_current.present? || removed_by_current.present?,
changed_by_me: added_by_me.present? || removed_by_me.present?,
conflict: set_snapshot_conflict?(added_by_current:,
removed_by_current:,
added_by_me:,
removed_by_me:) }
end
def set_snapshot_conflict? added_by_current:, removed_by_current:,
added_by_me:, removed_by_me:
(added_by_current & removed_by_me).present? || (removed_by_current & added_by_me).present?
end
def apply_post_snapshot! post, snapshot
PostVersionRecorder.ensure_snapshot!(post, created_by_user: current_user)
post.update!(title: snapshot[:title],
video_ms: snapshot[:video_ms],
original_created_from: snapshot[:original_created_from],
original_created_before: snapshot[:original_created_before])
Tag.normalise_tags!(snapshot[:tag_names], with_tagme: false,
deny_deprecated: true,
with_sections: true) =>
{ tags: editable_tags, sections: }
TagVersioning.record_tag_snapshots!(editable_tags, created_by_user: current_user)
readonly_tags = post.tags.nico.to_a
tags = readonly_tags + editable_tags
tags = Tag.expand_parent_tags(tags).reject(&:deprecated?)
post.video_ms = tags.any? { _1.id == Tag.video.id } ? snapshot[:video_ms] : nil
validate_video_sections!(post.video_ms, sections)
post.save!
sync_post_tags!(post, tags, sections)
sync_parent_posts!(post, snapshot[:parent_post_ids])
PostVersionRecorder.record!(post:, event_type: :update, created_by_user: current_user)
end
def merge_post_snapshots base_snapshot, current_snapshot, incoming_snapshot
[:title, :video_ms, :original_created_from, :original_created_before].map {
[_1, merge_scalar_snapshot_value(base_snapshot[_1],
current_snapshot[_1],
incoming_snapshot[_1])]
}.to_h.merge([:tag_names, :parent_post_ids].map {
[_1, merge_set_snapshot_value(base_snapshot[_1],
current_snapshot[_1],
incoming_snapshot[_1])]
}.to_h)
end
def merge_scalar_snapshot_value base, current, mine
return mine if current == base
return current if mine == base || current == mine
raise ArgumentError, '競合してゐる項目はマージできません.'
end
def merge_set_snapshot_value base, current, mine
base = base.to_a
current = current.to_a
mine = mine.to_a
added_by_current = current - base
removed_by_current = base - current
added_by_me = mine - base
removed_by_me = base - mine
merged = base + added_by_current + added_by_me
merged -= removed_by_current
merged -= removed_by_me
merged.uniq.sort
end
def render_post_form_record_invalid record
if record.is_a?(TagName) || record.is_a?(Tag)
render_validation_error fields: { tags: record.errors.full_messages.map { |message|
"タグ名 “#{ record.name }”: #{ message }"
} }
else
render_validation_error record
end
end
def normalise_video_ms tags, video_ms_param: params[:video_ms], duration_param: params[:duration]
return nil unless tags.any? { _1.id == Tag.video.id }
if video_ms_param.present?
video_ms = Integer(video_ms_param, exception: false)
raise VideoMsParseError unless video_ms&.positive?
return video_ms
end
return nil if duration_param.blank?
video_ms = Tag.time_to_ms!(duration_param.to_s, tag_name: '動画時間')
raise VideoMsParseError unless video_ms.positive?
video_ms
rescue Tag::SectionLiteralParseError
raise VideoMsParseError
end
def validate_video_sections! video_ms, sections
return unless video_ms
sections.each_value do |ranges|
ranges.each do |begin_ms, end_ms|
if begin_ms >= video_ms
post = Post.new
post.errors.add :video_ms, 'タグ区間の開始が動画時間以上です.'
raise ActiveRecord::RecordInvalid, post
end
if end_ms && end_ms > video_ms
post = Post.new
post.errors.add :video_ms, 'タグ区間の終端が動画時間を超えてゐます.'
raise ActiveRecord::RecordInvalid, post
end
end
end
end
end
+43 -34
ファイルの表示
@@ -1,46 +1,55 @@
class PreviewController < ApplicationController
before_action :require_member!
def title
# TODO: # 既知サイトなら決まったフォーマットで title 取得するやぅに.
return head :unauthorized unless current_user
return render_bad_request('URL は必須です.') if params[:url].blank?
url = params[:url]
return head :bad_request unless url.present?
unless url.start_with?(/http(s)?:\/\//)
url = 'http://' + url
end
html = URI.open(url, open_timeout: 5, read_timeout: 5).read
doc = Nokogiri::HTML.parse(html)
title = doc.at('title')&.text&.strip
render json: { title: title }
rescue => e
render json: { error: e.message }, status: :bad_request
render json: { title: Preview::ThumbnailFetcher.title(params[:url]) }
rescue Preview::UrlSafety::UnsafeUrl => e
render_bad_request(e.message)
rescue Preview::HttpFetcher::FetchTimeout => e
render_preview_error(e.message, :gateway_timeout)
rescue Preview::HttpFetcher::ResponseTooLarge => e
render_preview_error(e.message, :payload_too_large)
rescue Preview::HttpFetcher::FetchFailed => e
render_preview_error(e.message, :bad_gateway)
end
def thumbnail
# TODO: 既知ドメインであれば指定のアドレスからサムネールを取得するやぅにする.
return render_bad_request('URL は必須です.') if params[:url].blank?
attachment =
Post.resized_thumbnail_attachment(
StringIO.new(Preview::ThumbnailFetcher.fetch(params[:url])))
send_data attachment[:io].read,
type: attachment[:content_type],
disposition: 'inline'
rescue Preview::UrlSafety::UnsafeUrl => e
render_bad_request(e.message)
rescue Preview::HttpFetcher::FetchTimeout => e
render_preview_error(e.message, :gateway_timeout)
rescue Preview::HttpFetcher::ResponseTooLarge => e
render_preview_error(e.message, :payload_too_large)
rescue Preview::HttpFetcher::FetchFailed => e
render_preview_error(e.message, :bad_gateway)
rescue Preview::ThumbnailFetcher::GenerationFailed, MiniMagick::Error => e
render_unprocessable_entity(e.message)
end
private
def require_member!
return head :unauthorized unless current_user
return if current_user.gte_member?
url = params[:url]
return head :bad_request if url.blank?
head :forbidden
end
unless url.start_with?(/http(s)?:\/\//)
url = 'http://' + url
end
path = Rails.root.join('tmp', "thumb_#{ SecureRandom.hex }.png")
system("node #{ Rails.root }/lib/screenshot.js #{ Shellwords.escape(url) } #{ path }")
if File.exist?(path)
image = MiniMagick::Image.open(path)
image.resize '180x180'
File.delete(path) rescue nil
send_file image.path, type: 'image/png', disposition: 'inline'
else
render json: { error: 'Failed to generate thumbnail' }, status: :internal_server_error
end
def render_preview_error(message, status)
render json: { type: status.to_s,
message:,
errors: { },
base_errors: [message] },
status:
end
end
+24 -4
ファイルの表示
@@ -5,9 +5,19 @@ class TagChildrenController < ApplicationController
parent_id = params[:parent_id]
child_id = params[:child_id]
return head :bad_request if parent_id.blank? || child_id.blank?
return render_bad_request('parent_id は必須です.') if parent_id.blank?
return render_bad_request('child_id は必須です.') if child_id.blank?
Tag.find(parent_id).children << Tag.find(child_id) rescue nil
parent = Tag.find(parent_id)
child = Tag.find(child_id)
return render_bad_request('ニコニコ・タグの階層は変更できません.') if parent.nico? || child.nico?
ApplicationRecord.transaction do
TagVersioning.ensure_snapshot!(child, created_by_user: current_user)
TagImplication.find_or_create_by!(parent_tag: parent, tag: child)
TagVersionRecorder.record!(tag: child, event_type: :update, created_by_user: current_user)
end
head :no_content
end
@@ -18,9 +28,19 @@ class TagChildrenController < ApplicationController
parent_id = params[:parent_id]
child_id = params[:child_id]
return head :bad_request if parent_id.blank? || child_id.blank?
return render_bad_request('parent_id は必須です.') if parent_id.blank?
return render_bad_request('child_id は必須です.') if child_id.blank?
Tag.find(parent_id).children.delete(Tag.find(child_id)) rescue nil
parent = Tag.find(parent_id)
child = Tag.find(child_id)
return render_bad_request('ニコニコ・タグの階層は変更できません.') if parent.nico? || child.nico?
ApplicationRecord.transaction do
TagVersioning.ensure_snapshot!(child, created_by_user: current_user)
TagImplication.find_by(parent_tag: parent, tag: child)&.destroy!
TagVersionRecorder.record!(tag: child, event_type: :update, created_by_user: current_user)
end
head :no_content
end
+95
ファイルの表示
@@ -0,0 +1,95 @@
class TagVersionsController < ApplicationController
def index
tag_id = params[:id].presence
page = (params[:page].presence || 1).to_i
limit = (params[:limit].presence || 20).to_i
page = 1 if page < 1
limit = 1 if limit < 1
offset = (page - 1) * limit
q = TagVersion.joins(<<~SQL.squish)
LEFT JOIN
tag_versions prev
ON
prev.tag_id = tag_versions.tag_id
AND prev.version_no = tag_versions.version_no - 1
SQL
.select('tag_versions.*', 'prev.name AS prev_name', 'prev.category AS prev_category',
'prev.deprecated_at AS prev_deprecated_at',
'prev.aliases AS prev_aliases', 'prev.parent_tag_ids AS prev_parent_tag_ids')
q = q.where('tag_versions.tag_id = ?', tag_id) if tag_id
count = q.except(:select, :order, :limit, :offset).count
versions = q.order(Arel.sql('tag_versions.created_at DESC, tag_versions.id DESC'))
.limit(limit)
.offset(offset)
render json: { versions: serialise_versions(versions), count: }
end
private
def serialise_versions rows
user_ids = rows.map(&:created_by_user_id).compact.uniq
users_by_id = User.where(id: user_ids).pluck(:id, :name).to_h
rows.map do |row|
cur_aliases = split_values(row.aliases)
prev_aliases = split_values(row.attributes['prev_aliases'])
cur_parent_tag_ids = split_parent_tag_ids(row.parent_tag_ids)
prev_parent_tag_ids = split_parent_tag_ids(row.attributes['prev_parent_tag_ids'])
all_parent_tag_ids = (cur_parent_tag_ids | prev_parent_tag_ids)
tags_by_id =
Tag
.includes(:tag_name, :materials, { tag_name: :wiki_page })
.where(id: all_parent_tag_ids)
.index_by(&:id)
parent_tags =
build_version_values(cur_parent_tag_ids, prev_parent_tag_ids, key: :tag_id)
.map do |h|
{ tag: TagRepr.base(tags_by_id[h[:tag_id]]),
type: h[:type] }
end
{ tag_id: row.tag_id,
version_no: row.version_no,
event_type: row.event_type,
name: { current: row.name, prev: row.attributes['prev_name'] },
category: { current: row.category, prev: row.attributes['prev_category'] },
deprecated_at: { current: row.deprecated_at&.iso8601,
prev: row.attributes['prev_deprecated_at']&.iso8601 },
aliases: build_version_values(cur_aliases, prev_aliases, key: :name),
parent_tags:,
created_at: row.created_at.iso8601,
created_by_user: row.created_by_user_id &&
{ id: row.created_by_user_id,
name: users_by_id[row.created_by_user_id] } }
end
end
def build_version_values cur_values, prev_values, key:
(cur_values | prev_values).map do |value|
type =
if cur_values.include?(value) && prev_values.include?(value)
'context'
elsif cur_values.include?(value)
'added'
else
'removed'
end
{ key => value, type: }
end
end
def split_values(values) = values.to_s.split(/\s+/).reject(&:blank?)
def split_parent_tag_ids(values) = split_values(values).map(&:to_i)
end
+529 -23
ファイルの表示
@@ -1,15 +1,22 @@
require 'net/http'
require 'uri'
require 'set'
class TagsController < ApplicationController
def index
post_id = params[:post]
name = params[:name].to_s.strip.presence
category = params[:category].to_s.strip.presence
name = params[:name].presence
category = params[:category].presence
post_count_between = (params[:post_count_gte].presence || -1).to_i,
(params[:post_count_lte].presence || -1).to_i
post_count_between[0] = nil if post_count_between[0] < 0
post_count_between[1] = nil if post_count_between[1] < 0
created_between = params[:created_from].presence, params[:created_to].presence
updated_between = params[:updated_from].presence, params[:updated_to].presence
deprecated_given = params.key?(:deprecated)
deprecated = bool?(:deprecated)
order = params[:order].to_s.split(':', 2).map(&:strip)
unless order[0].in?(['name', 'category', 'post_count', 'created_at', 'updated_at'])
@@ -33,19 +40,20 @@ class TagsController < ApplicationController
else
Tag.joins(:tag_name)
end
.includes(:tag_name, tag_name: :wiki_page)
.includes(:tag_name, :materials, tag_name: :wiki_page)
q = q.where(posts: { id: post_id }) if post_id.present?
if name
q = q.where('tag_names.name LIKE ?', "%#{ ActiveRecord::Base.sanitize_sql_like(name) }%")
end
q = q.where(category: category) if category
q = q.where('tag_names.name LIKE ?', "%#{ name }%") if name
q = q.where(category:) if category
q = q.where('tags.post_count >= ?', post_count_between[0]) if post_count_between[0]
q = q.where('tags.post_count <= ?', post_count_between[1]) if post_count_between[1]
q = q.where('tags.created_at >= ?', created_between[0]) if created_between[0]
q = q.where('tags.created_at <= ?', created_between[1]) if created_between[1]
q = q.where('tags.updated_at >= ?', updated_between[0]) if updated_between[0]
q = q.where('tags.updated_at <= ?', updated_between[1]) if updated_between[1]
if deprecated_given
q = deprecated ? q.where.not(deprecated_at: nil) : q.where(deprecated_at: nil)
end
sort_sql =
case order[0]
@@ -68,7 +76,26 @@ class TagsController < ApplicationController
.offset(offset)
.to_a
render json: { tags: TagRepr.base(tags), count: q.size }
render json: { tags: TagRepr.many(tags), count: q.size }
end
def with_depth
parent_tag_id = params[:parent].to_i
parent_tag_id = nil if parent_tag_id <= 0
material_filter = material_filter_param(default: 'any')
graph = build_with_depth_graph(material_filter)
tag_ids =
if parent_tag_id
visible_child_tag_ids(parent_tag_id, graph)
else
visible_root_tag_ids(graph)
end
render json: tag_ids
.sort_by { |tag_id| graph[:tags_by_id][tag_id][:name] }
.map { |tag_id| with_depth_lightweight_repr(tag_id, graph) }
end
def autocomplete
@@ -79,7 +106,7 @@ class TagsController < ApplicationController
alias_rows =
TagName
.where('name LIKE ?', "#{ ActiveRecord::Base.sanitize_sql_like(q) }%")
.where('name LIKE ?', "#{ q }%")
.where.not(canonical_id: nil)
.pluck(:canonical_id, :name)
@@ -92,15 +119,15 @@ class TagsController < ApplicationController
end
base = Tag.joins(:tag_name)
.includes(:tag_name, tag_name: :wiki_page)
.includes(:tag_name, :materials, tag_name: :wiki_page)
.where(deprecated_at: nil)
base = base.where('tags.post_count > 0') if present_only
canonical_hit =
base
.where(((with_nico ? '(tags.category = ? AND tag_names.name LIKE ?) OR ' : '') +
'tag_names.name LIKE ?'),
*(with_nico ? ['nico', "nico:#{ ActiveRecord::Base.sanitise_sql_like(q) }%"] : []),
"#{ ActiveRecord::Base.sanitise_sql_like(q) }%")
*(with_nico ? ['nico', "nico:#{ q }%"] : []), "#{ q }%")
tags =
if canonical_ids.present?
@@ -118,7 +145,7 @@ class TagsController < ApplicationController
def show
tag = Tag.joins(:tag_name)
.includes(:tag_name, tag_name: :wiki_page)
.includes(:tag_name, :materials, tag_name: :wiki_page)
.find_by(id: params[:id])
if tag
render json: TagRepr.base(tag)
@@ -129,10 +156,10 @@ class TagsController < ApplicationController
def show_by_name
name = params[:name].to_s.strip
return head :bad_request if name.blank?
return render_bad_request('name は必須です.') if name.blank?
tag = Tag.joins(:tag_name)
.includes(:tag_name, tag_name: :wiki_page)
.includes(:tag_name, :materials, tag_name: :wiki_page)
.find_by(tag_names: { name: })
if tag
render json: TagRepr.base(tag)
@@ -147,19 +174,122 @@ class TagsController < ApplicationController
.find_by(id: params[:id])
return head :not_found unless tag
render json: DeerjikistRepr.many(tag.deerjikists)
render json: { tag: TagRepr.base(tag),
deerjikists: DeerjikistRepr.many(tag.deerjikists) }
end
def deerjikists_by_name
name = params[:name].to_s.strip
return head :bad_request if name.blank?
return render_bad_request('name は必須です.') if name.blank?
tag = Tag.joins(:tag_name)
.includes(:tag_name, tag_name: :wiki_page)
.find_by(tag_names: { name: })
return head :not_found unless tag
render json: DeerjikistRepr.many(tag.deerjikists)
render json: { tag: TagRepr.base(tag),
deerjikists: DeerjikistRepr.many(tag.deerjikists) }
end
def update_deerjikists
return head :unauthorized unless current_user
return head :forbidden unless current_user.gte_member?
tag = Tag.joins(:tag_name)
.includes(:tag_name, tag_name: :wiki_page)
.find_by(id: params[:id])
return head :not_found unless tag
ApplicationRecord.transaction do
tag.deerjikists = []
params[:_json].each.with_index do |item, i|
platform = item[:platform]
code = normalise_deerjikist_code(platform, item[:code])
deerjikist = Deerjikist.find_or_initialize_by(platform:, code:)
deerjikist.tag = tag
render_deerjikist_form_record_invalid(deerjikist, i) unless deerjikist.save
raise ActiveRecord::Rollback if performed?
end
end
return if performed?
render json: DeerjikistRepr.many(tag.reload.deerjikists)
end
def materials_by_name
name = params[:name].to_s.strip
return render_bad_request('name は必須です.') if name.blank?
material_filter = material_filter_param(default: 'any')
tag = Tag.joins(:tag_name)
.includes(:tag_name, :materials, tag_name: :wiki_page)
.find_by(tag_names: { name: })
return head :not_found unless tag
graph = build_with_depth_graph(material_filter)
render json: build_tag_children(tag, graph:)
end
def update_all
return head :unauthorized unless current_user
return head :forbidden unless current_user.gte_member?
tag = Tag.find_by(id: params[:id])
return head :not_found unless tag
name = params[:name].to_s.strip
category = params[:category].to_s.strip
return render_unprocessable_entity('名前は必須です.', field: :name) if name.blank?
return render_unprocessable_entity('カテゴリは必須です.', field: :category) if category.blank?
return render_unprocessable_entity '廃止状態は必須です.', field: :deprecated unless params.key?(:deprecated)
return unless validate_tag_rename(tag, name)
alias_names = params[:aliases].to_s.split.uniq
parent_names = params[:parent_tags].to_s.split.uniq
deprecated = bool?(:deprecated)
if tag.nico? && deprecated
return render_unprocessable_entity 'ニコタグは廃止できません.', field: :deprecated
end
if tag.nico? || category == 'nico'
return render_unprocessable_entity 'ニコタグは変更できません.', field: :category
end
ApplicationRecord.transaction do
TagVersioning.ensure_snapshot!(tag, created_by_user: current_user)
old_name = tag.name
name_changed = name != old_name
wiki_page = tag.tag_name.wiki_page if name_changed
if tag.deprecated? == deprecated
tag.update!(category:)
else
tag.update!(category:, deprecated_at: deprecated ? Time.current : nil)
end
rename_tag_name!(tag, name) if name_changed
alias_names << old_name if name_changed
alias_names.delete(name)
update_aliases!(tag, alias_names)
update_parent_tags!(tag, parent_names)
tag.reload
record_tag_version!(
tag,
event_type: :update,
created_by_user: current_user,
name_changed:,
wiki_page:)
end
render json: TagRepr.base(tag.reload)
end
def update
@@ -168,17 +298,393 @@ class TagsController < ApplicationController
name = params[:name].presence
category = params[:category].presence
deprecated_given = params.key?(:deprecated)
deprecated = bool?(:deprecated)
tag = Tag.find(params[:id])
if name.present?
tag.tag_name.update!(name:)
if tag.nico? && deprecated_given && deprecated
return render_unprocessable_entity 'ニコタグは廃止できません.', field: :deprecated
end
if category.present?
tag.update!(category:)
return unless validate_tag_rename(tag, name)
if tag.nico? || (category.present? && category == 'nico')
return render_unprocessable_entity 'ニコタグは変更できません.', field: :category
end
render json: TagRepr.base(tag)
ApplicationRecord.transaction do
TagVersioning.ensure_snapshot!(tag, created_by_user: current_user)
old_name = tag.name
name_changed = name.present? && name != old_name
wiki_page = tag.tag_name.wiki_page if name_changed
rename_tag_name!(tag, name) if name_changed
tag.update!(category:) if category.present?
if deprecated_given && tag.deprecated? != deprecated
tag.update!(deprecated_at: deprecated ? Time.current : nil)
end
tag.reload
record_tag_version!(
tag,
event_type: :update,
created_by_user: current_user,
name_changed:,
wiki_page:)
end
render json: TagRepr.base(tag.reload)
end
private
def material_filter_param default:
value = params[:material_filter].to_s.presence
return default unless ['present', 'missing', 'any'].include?(value)
value
end
def build_with_depth_graph material_filter
children_by_parent_id = Hash.new { |h, k| h[k] = [] }
parent_ids_by_child_id = Hash.new { |h, k| h[k] = [] }
TagImplication.pluck(:parent_tag_id, :tag_id).each do |parent_id, child_id|
children_by_parent_id[parent_id] << child_id
parent_ids_by_child_id[child_id] << parent_id
end
tag_ids = (children_by_parent_id.keys +
parent_ids_by_child_id.keys +
Tag.where(category: ['meme', 'character', 'material']).pluck(:id)).uniq
material_tag_ids = Material.unscoped.kept.where.not(tag_id: nil).distinct.pluck(:tag_id).to_set
tags_by_id = Tag.joins(:tag_name)
.where(id: tag_ids)
.pluck('tags.id', 'tag_names.name', 'tags.category', 'tags.deprecated_at')
.each_with_object({ }) do |(id, name, category, deprecated_at), h|
h[id] = { name:, category:, deprecated: deprecated_at.present?,
has_material: material_tag_ids.include?(id) }
end
{ children_by_parent_id:, parent_ids_by_child_id:, tags_by_id:,
visible_child_tag_ids_by_parent_id: { },
visible_subtree_by_tag_id: { },
material_filter: }
end
def visible_root_tag_ids graph
root_ids = Set.new
graph[:tags_by_id].each do |tag_id, attrs|
next unless visible_root_tag?(tag_id, graph)
next unless visible_subtree?(tag_id, graph)
if attrs[:deprecated]
collect_visible_child_tag_ids(tag_id, graph, root_ids, Set.new)
else
root_ids << tag_id
end
end
root_ids.to_a
end
def visible_root_tag? tag_id, graph
seen = Set.new([tag_id])
stack = graph[:parent_ids_by_child_id][tag_id].dup
until stack.empty?
parent_id = stack.pop
next if seen.include?(parent_id)
seen << parent_id
parent = graph[:tags_by_id][parent_id]
next unless parent
return false unless parent[:deprecated]
stack.concat(graph[:parent_ids_by_child_id][parent_id])
end
true
end
def visible_child_tag_ids parent_tag_id, graph
cache = graph[:visible_child_tag_ids_by_parent_id]
return cache[parent_tag_id] if cache.key?(parent_tag_id)
visible_ids = Set.new
graph[:children_by_parent_id][parent_tag_id].each do |child_tag_id|
collect_visible_child_tag_ids(child_tag_id, graph, visible_ids, Set.new([parent_tag_id]))
end
cache[parent_tag_id] = visible_ids.to_a
end
def collect_visible_child_tag_ids tag_id, graph, visible_ids, seen
return if seen.include?(tag_id)
seen = seen.dup << tag_id
tag = graph[:tags_by_id][tag_id]
return unless tag
if tag[:deprecated]
graph[:children_by_parent_id][tag_id].each do |child_tag_id|
collect_visible_child_tag_ids(child_tag_id, graph, visible_ids, seen)
end
return
end
if with_depth_visible_tag?(tag, graph[:material_filter])
visible_ids << tag_id
return
end
visible_ids << tag_id if graph[:children_by_parent_id][tag_id].any? { |child_tag_id|
visible_subtree?(child_tag_id, graph)
}
end
def with_depth_visible_tag? tag, material_filter
return false unless tag[:category].in?(['meme', 'character', 'material']) && !tag[:deprecated]
case material_filter
when 'present'
tag[:has_material]
when 'missing'
!tag[:has_material]
else
true
end
end
def visible_subtree? tag_id, graph
cache = graph[:visible_subtree_by_tag_id]
return cache[tag_id] if cache.key?(tag_id)
tag = graph[:tags_by_id][tag_id]
return cache[tag_id] = false unless tag
if tag[:deprecated]
return cache[tag_id] =
graph[:children_by_parent_id][tag_id].any? { |child_tag_id|
visible_subtree?(child_tag_id, graph)
}
end
cache[tag_id] =
with_depth_visible_tag?(tag, graph[:material_filter]) ||
graph[:children_by_parent_id][tag_id].any? { |child_tag_id|
visible_subtree?(child_tag_id, graph)
}
end
def with_depth_lightweight_repr tag_id, graph
tag = graph[:tags_by_id][tag_id]
{ id: tag_id,
name: tag[:name],
category: tag[:category],
deprecated: tag[:deprecated],
has_material: tag[:has_material],
has_children: visible_child_tag_ids(tag_id, graph).present?,
children: [] }
end
def build_tag_children tag, graph: nil
material = tag.materials.first
tag_graph = graph && graph[:tags_by_id][tag.id]
material =
nil if tag_graph && !with_depth_visible_tag?(tag_graph, graph[:material_filter])
children = tag.children.sort_by(&:name)
if graph
children = children.filter { |child_tag| visible_subtree?(child_tag.id, graph) }
end
TagRepr.base(tag).merge(
children: children.map { build_tag_children(_1, graph:) },
has_material: tag_graph ? tag_graph[:has_material] : material.present?,
material: material && MaterialRepr.base(material, host: request.base_url))
end
def record_tag_version! tag, event_type:, created_by_user:, name_changed: false, wiki_page: nil
if tag.nico?
NicoTagVersionRecorder.record!(tag:, event_type:, created_by_user:)
return
end
TagVersionRecorder.record!(tag:, event_type:, created_by_user:)
return unless name_changed
wiki_page ||= tag.tag_name.wiki_page
return unless wiki_page&.wiki_versions&.exists?
WikiVersionRecorder.record!(
page: wiki_page,
event_type: :update,
created_by_user:)
end
def validate_tag_rename tag, name
return true if name.blank? || name == tag.name
if tag.in?([Tag.tagme, Tag.bot, Tag.no_deerjikist, Tag.video, Tag.niconico])
render_unprocessable_entity 'システム・タグの名称は変更できません.', field: :name
return false
end
target_tag_name = TagName.with_discarded.find_by(name:)
return true if target_tag_name.nil?
return true if target_tag_name.canonical_id?
render_unprocessable_entity 'その名前は既に使はれてゐます.', field: :name
false
end
def rename_tag_name! tag, name
return if name == tag.name
current_tag_name = tag.tag_name
target_tag_name = TagName.with_discarded.find_by(name:)
if target_tag_name.nil?
current_tag_name.update!(name:)
return
end
promote_tag_alias!(
tag,
current_tag_name:,
promoted_tag_name: target_tag_name)
end
def promote_tag_alias! tag, current_tag_name:, promoted_tag_name:
old_owner_tag = promoted_tag_name.canonical&.tag
if old_owner_tag && old_owner_tag != tag
TagVersioning.ensure_snapshot!(old_owner_tag, created_by_user: current_user)
end
promoted_tag_name.undiscard! if promoted_tag_name.discarded?
promoted_tag_name.update!(canonical: nil)
TagName.with_discarded
.where(canonical_id: current_tag_name.id)
.where.not(id: promoted_tag_name.id)
.find_each do |alias_tag_name|
alias_tag_name.update!(canonical: promoted_tag_name)
end
current_tag_name.wiki_page&.update!(tag_name: promoted_tag_name)
tag.update!(tag_name: promoted_tag_name)
current_tag_name.association(:wiki_page).reset
current_tag_name.association(:tag).reset
current_tag_name.reload.update!(canonical: promoted_tag_name)
return unless old_owner_tag && old_owner_tag != tag
record_tag_version!(old_owner_tag.reload, event_type: :update, created_by_user: current_user)
end
def update_aliases! tag, alias_names
alias_names = alias_names.uniq
affected_tags = [tag]
current_aliases = tag.tag_name.aliases.to_a
current_aliases.each do |alias_tag_name|
next if alias_names.include?(alias_tag_name.name)
affected_tags << alias_tag_name.canonical&.tag
end
alias_names.each do |alias_name|
alias_tag_name = TagName.find_undiscard_or_create_by!(name: alias_name)
affected_tags << alias_tag_name.canonical&.tag
end
affected_tags.compact.uniq.each do |affected_tag|
TagVersioning.ensure_snapshot!(affected_tag, created_by_user: current_user)
end
current_aliases.each do |alias_tag_name|
next if alias_names.include?(alias_tag_name.name)
alias_tag_name.update!(canonical: nil)
end
alias_names.each do |alias_name|
alias_tag_name = TagName.find_undiscard_or_create_by!(name: alias_name)
alias_tag_name.update!(canonical: tag.tag_name)
end
affected_tags.compact.uniq.each do |affected_tag|
record_tag_version!(affected_tag, event_type: :update, created_by_user: current_user)
end
end
def update_parent_tags! tag, parent_names
parent_tags = Tag.normalise_tags!(parent_names, with_tagme: false,
with_no_deerjikist: false,
deny_nico: true)
old_parent_tags = tag.parents.to_a
TagVersioning.record_tag_snapshots!((old_parent_tags + parent_tags).uniq,
created_by_user: current_user)
tag.reversed_tag_implications.destroy_all
parent_tags.each do |parent_tag|
next if parent_tag == tag
TagImplication.create!(tag:, parent_tag:)
end
end
def normalise_deerjikist_code platform, code
return code if platform != 'youtube' || code[0] != '@'
url = "https://www.youtube.com/#{ code }"
html = Net::HTTP.get(URI(url))
canonical = html[
/<link rel="canonical" href="https:\/\/www\.youtube\.com\/channel\/(UC[a-zA-Z0-9_-]{22})"/,
1]
return canonical if canonical
html[/"channelId":"(UC[a-zA-Z0-9_-]{22})"/, 1] || html[/\bUC[a-zA-Z0-9_-]{22}\b/]
rescue
nil
end
def render_deerjikist_form_record_invalid deerjikist, index
fields = { }
deerjikist.errors.each do |error|
field =
case error.attribute
when :platform, :code
"deerjikists.#{ index }.#{ error.attribute }"
else
:deerjikists
end
fields[field] ||= []
fields[field] << error.full_message
end
render_validation_error fields:
end
end
+54
ファイルの表示
@@ -0,0 +1,54 @@
class TheatreCommentsController < ApplicationController
def index
limit = params[:limit].to_i
limit = 20 if limit <= 0
no_gt = params[:no_gt].to_i
no_gt = 0 if no_gt < 0
comments = TheatreComment
.where(theatre_id: params[:theatre_id])
.where('no > ?', no_gt)
.order(no: :desc)
.limit(limit)
render json: comments.map {
_1.as_json(include: { user: { only: [:id, :name] } })
.merge(content: _1.discarded? ? nil : _1.content, deleted: _1.discarded?)
}
end
def create
return head :unauthorized unless current_user
content = params[:content]
return render_unprocessable_entity('本文は必須です.', field: :content) if content.blank?
theatre = Theatre.find_by(id: params[:theatre_id])
return head :not_found unless theatre
comment = nil
theatre.with_lock do
no = theatre.next_comment_no
comment = TheatreComment.create!(theatre:, no:, user: current_user, content:)
theatre.update!(next_comment_no: no + 1)
end
render json: comment, status: :created
end
def destroy
return head :unauthorized unless current_user
theatre_id = params[:theatre_id].to_i
no = params[:id].to_i
comment = TheatreComment.find_by(theatre_id:, no:)
return head :not_found unless comment
return head :forbidden unless comment.user == current_user
comment.discard!
head :no_content
end
end
+22
ファイルの表示
@@ -0,0 +1,22 @@
class TheatreProgrammesController < ApplicationController
def index
limit = params[:limit].to_i
limit = 100 if limit <= 0
position_gt = params[:position_gt].to_i
position_gt = 0 if position_gt < 0
programmes = TheatreProgramme
.where(theatre_id: params[:theatre_id])
.where('position > ?', position_gt)
.includes(:post)
.order(position: :desc).limit(100)
.limit(limit)
render json: programmes.map { |programme|
programme.as_json.merge(post: { id: programme.post.id,
title: programme.post.title,
url: programme.post.url })
}
end
end
+22
ファイルの表示
@@ -0,0 +1,22 @@
class TheatreSkipEventsController < ApplicationController
def index
limit = params[:limit].to_i
limit = 50 if limit <= 0
events =
TheatreSkipEvent
.where(theatre_id: params[:theatre_id])
.includes(:post, tags: :tag_name)
.order(created_at: :desc)
.limit(limit)
render json: events.map { |event|
{ id: event.id,
theatre_id: event.theatre_id,
post: { id: event.post.id, title: event.post.title, url: event.post.url },
tags: event.tags.map { |tag| { id: tag.id, name: tag.name } },
programme_position: event.programme_position,
created_at: event.created_at }
}
end
end
+113 -6
ファイルの表示
@@ -31,7 +31,7 @@ class TheatresController < ApplicationController
post_started_at = theatre.current_post_started_at
end
render json: { host_flg:, post_id:, post_started_at: }
render json: theatre_info_json(theatre, host_flg:, post_id:, post_started_at:)
end
def next_post
@@ -41,12 +41,119 @@ class TheatresController < ApplicationController
return head :not_found unless theatre
return head :forbidden if theatre.host_user != current_user
post = Post.where("url LIKE '%nicovideo.jp%'")
.or(Post.where("url LIKE '%youtube.com%'"))
.order('RAND()')
.first
theatre.update!(current_post: post, current_post_started_at: Time.current)
ApplicationRecord.transaction do
theatre.lock!
TheatrePostAdvancer.call(theatre:)
end
head :no_content
end
def skip_vote
return head :unauthorized unless current_user
theatre = Theatre.find_by(id: params[:id])
return head :not_found unless theatre
requested_post_id = params[:post_id].to_i
return head :unprocessable_entity if requested_post_id <= 0
skipped = false
conflicted = false
ApplicationRecord.transaction do
theatre.lock!
if theatre.current_post
TheatreWatchingUser.find_or_initialize_by(theatre:, user: current_user).tap {
_1.expires_at = 30.seconds.from_now
}.save!
if theatre.current_post_id != requested_post_id
conflicted = true
next
end
TheatreSkipVote.find_or_create_by!(theatre:, post_id: requested_post_id, user: current_user)
vote_status = skip_vote_status(theatre)
if vote_status[:votes_count] >= vote_status[:required_count]
TheatreSkipFinalizer.call(theatre:, user: current_user)
TheatrePostAdvancer.call(theatre:)
skipped = true
end
end
end
theatre.reload
return render json: theatre_info_json(theatre, skipped: false), status: :conflict if conflicted
render json: theatre_info_json(theatre, skipped:)
end
def unskip_vote
return head :unauthorized unless current_user
theatre = Theatre.find_by(id: params[:id])
return head :not_found unless theatre
requested_post_id = params[:post_id].to_i
return head :unprocessable_entity if requested_post_id <= 0
conflicted = false
theatre.with_lock do
if theatre.current_post
if theatre.current_post_id != requested_post_id
conflicted = true
else
TheatreSkipVote.where(theatre:, post_id: requested_post_id, user: current_user).delete_all
end
end
end
theatre.reload
return render json: theatre_info_json(theatre, skipped: false), status: :conflict if conflicted
render json: theatre_info_json(theatre, skipped: false)
end
def post_selection_weights
theatre = Theatre.find_by(id: params[:id])
return head :not_found unless theatre
render json: TheatrePostSelector.new(theatre:).weight_json
end
private
def theatre_info_json(theatre, host_flg: nil, post_id: nil, post_started_at: nil, skipped: nil)
host_flg = theatre.host_user_id == current_user&.id if host_flg.nil?
post_id = theatre.current_post_id if post_id.nil?
post_started_at = theatre.current_post_started_at if post_started_at.nil?
json = { host_flg:,
post_id:,
post_started_at:,
post_elapsed_ms: post_started_at ? ((Time.current - post_started_at) * 1000).floor : nil,
watching_users: theatre.watching_users.as_json(only: [:id, :name]),
skip_vote: skip_vote_status(theatre) }
json[:skipped] = skipped unless skipped.nil?
json
end
def skip_vote_status(theatre)
watching_user_ids = theatre.watching_users.ids
watching_users_count = watching_user_ids.size
required_count = (watching_users_count / 2) + 1
post = theatre.current_post
votes =
if post
TheatreSkipVote.where(theatre:, post:, user_id: watching_user_ids)
else
TheatreSkipVote.none
end
{ votes_count: post ? votes.count : 0,
required_count:,
watching_users_count:,
voted: post && current_user ? votes.exists?(user_id: current_user.id) : false }
end
end
+61
ファイルの表示
@@ -0,0 +1,61 @@
class UserSettingsController < ApplicationController
wrap_parameters false
def show
return head :unauthorized unless current_user
render json: current_setting.serializable_hash
end
def update
return head :unauthorized unless current_user
raw_attributes = editable_raw_attributes
field_errors = validate_raw_attributes(raw_attributes)
return render_validation_error fields: field_errors if field_errors.present?
setting = current_setting
setting.assign_attributes(raw_attributes)
if setting.save
render json: setting.serializable_hash, status: :ok
else
render_validation_error setting
end
end
private
def current_setting
Setting.find_or_create_by!(user: current_user) do |setting|
setting.assign_attributes(Setting.defaults)
end
rescue ActiveRecord::RecordNotUnique
Setting.find_by!(user: current_user)
end
def editable_raw_attributes
request.request_parameters.slice(*Setting::EDITABLE_ATTRIBUTES)
end
def validate_raw_attributes raw_attributes
raw_attributes.each_with_object({ }) do |(key, value), errors|
next if value_matches_type?(key, value)
errors[key.to_sym] = ['値の型が不正です.']
end
end
def value_matches_type? key, value
case Setting::TYPE_BY_ATTRIBUTE.fetch(key)
when :string
value.is_a?(String)
when :integer
value.is_a?(Integer)
when :boolean
value == true || value == false
else
false
end
end
end
+38
ファイルの表示
@@ -0,0 +1,38 @@
class UserThemeSlotsController < ApplicationController
wrap_parameters false
def index
return head :unauthorized unless current_user
render json: current_user
.theme_slots
.order(:base_theme, :slot_no)
.map { |slot| slot.serializable_hash }
end
def update
return head :unauthorized unless current_user
base_theme = params[:base_theme].to_s
slot_no = params[:slot_no].to_i
tokens = params[:tokens]
unless UserThemeSlot::BASE_THEMES.include?(base_theme)
return render_validation_error fields: { base_theme: ['値が不正です.'] }
end
unless UserThemeSlot::SLOT_NOS.include?(slot_no)
return render_validation_error fields: { slot_no: ['値が不正です.'] }
end
unless tokens.is_a?(ActionController::Parameters) || tokens.is_a?(Hash)
return render_validation_error fields: { tokens: ['JSON object で指定してください.'] }
end
slot = UserThemeSlot.find_or_initialize_by(user: current_user,
base_theme:,
slot_no:)
slot.tokens = tokens.is_a?(ActionController::Parameters) ? tokens.to_unsafe_h : tokens
slot.save!
render json: slot.serializable_hash, status: :ok
end
end
+22 -9
ファイルの表示
@@ -1,18 +1,22 @@
class UsersController < ApplicationController
def create
user = User.create!(inheritance_code: SecureRandom.uuid, role: 'guest')
user = nil
User.transaction do
user = User.create!(inheritance_code: SecureRandom.uuid, role: :guest)
attach_ip_address!(user)
end
render json: { code: user.inheritance_code,
user: user.slice(:id, :name, :inheritance_code, :role) }
user: user.slice(:id, :name, :inheritance_code, :role) },
status: :created
end
def verify
ip_bin = IPAddr.new(request.remote_ip).hton
ip_address = IpAddress.find_or_create_by!(ip_address: ip_bin)
user = User.find_by(inheritance_code: params[:code])
return render json: { valid: false } unless user
return head :forbidden if user.banned?
UserIp.find_or_create_by!(user:, ip_address:)
attach_ip_address!(user)
render json: { valid: true, user: user.slice(:id, :name, :inheritance_code, :role) }
end
@@ -38,12 +42,21 @@ class UsersController < ApplicationController
return head :unauthorized if user&.id != params[:id].to_i
name = params[:name]
return head :bad_request if name.blank?
return render_unprocessable_entity('名前は必須です.', field: :name) if name.blank?
if user.update(name:)
render json: user.slice(:id, :name, :inheritance_code, :role), status: :created
render json: user.slice(:id, :name, :inheritance_code, :role), status: :ok
else
render json: user.errors, status: :unprocessable_entity
render_validation_error user
end
end
private
def attach_ip_address! user
ip_bin = IPAddr.new(request.remote_ip).hton
ip_address = IpAddress.create_or_find_by!(ip_address: ip_bin)
UserIp.create_or_find_by!(user:, ip_address:)
end
end
+57 -32
ファイルの表示
@@ -4,17 +4,18 @@ class WikiPagesController < ApplicationController
def index
title = params[:title].to_s.strip
if title.blank?
return render json: WikiPageRepr.base(WikiPage.joins(:tag_name).includes(:tag_name))
return render json: WikiPageRepr.base(
WikiPage.joins(:tag_name).includes(tag_name: :tag))
end
q = WikiPage.joins(:tag_name).includes(:tag_name)
q = WikiPage.joins(:tag_name).includes(tag_name: :tag)
.where('tag_names.name LIKE ?', "%#{ WikiPage.sanitize_sql_like(title) }%")
render json: WikiPageRepr.base(q.limit(20))
end
def show
page = WikiPage.joins(:tag_name)
.includes(:tag_name)
.includes(tag_name: :tag)
.find_by(id: params[:id])
render_wiki_page_or_404 page
end
@@ -22,7 +23,7 @@ class WikiPagesController < ApplicationController
def show_by_title
title = params[:title].to_s.strip
page = WikiPage.joins(:tag_name)
.includes(:tag_name)
.includes(tag_name: :tag)
.find_by(tag_name: { name: title })
render_wiki_page_or_404 page
end
@@ -46,17 +47,17 @@ class WikiPagesController < ApplicationController
def diff
id = params[:id]
return head :bad_request if id.blank?
return render_bad_request('id は必須です.') if id.blank?
from = params[:from].presence
to = params[:to].presence
page = WikiPage.joins(:tag_name).includes(:tag_name).find(id)
page = WikiPage.joins(:tag_name).includes(tag_name: :tag).find(id)
from_rev = from && page.wiki_revisions.find(from)
to_rev = to ? page.wiki_revisions.find(to) : page.current_revision
if ((from_rev && !(from_rev.content?)) || !(to_rev&.content?))
return head :unprocessable_entity
return render_unprocessable_entity('差分を表示できない版です.')
end
diffs = Diff::LCS.sdiff(from_rev&.body&.lines || [], to_rev.body.lines)
@@ -76,6 +77,7 @@ class WikiPagesController < ApplicationController
render json: { wiki_page_id: page.id,
title: page.title,
deprecated_at: page.deprecated_at,
older_revision_id: from_rev&.id,
newer_revision_id: to_rev.id,
diff: diff_json }
@@ -85,22 +87,27 @@ class WikiPagesController < ApplicationController
return head :unauthorized unless current_user
return head :forbidden unless current_user.gte_member?
name = params[:title]&.strip
title = params[:title].to_s.strip
body = params[:body].to_s
message = params[:message].presence
return head :unprocessable_entity if name.blank? || body.blank?
return render_unprocessable_entity('タイトルは必須です.', field: :title) if title.blank?
return render_unprocessable_entity('本文は必須です.', field: :body) if body.blank?
tag_name = TagName.find_undiscard_or_create_by!(name:)
page = WikiPage.new(tag_name:, created_user: current_user, updated_user: current_user)
if page.save
message = params[:message].presence
Wiki::Commit.content!(page:, body:, created_user: current_user, message:)
tag_name = TagName.find_undiscard_or_create_by!(name: title)
render json: WikiPageRepr.base(page), status: :created
else
render json: { errors: page.errors.full_messages },
status: :unprocessable_entity
end
page =
Wiki::Commit.create_content!(
tag_name:,
body:,
created_by_user: current_user,
message:)
render json: WikiPageRepr.base(page), status: :created
rescue ActiveRecord::RecordInvalid => e
render_validation_error e.record
rescue ActiveRecord::RecordNotUnique
render_record_not_unique
end
def update
@@ -110,22 +117,38 @@ class WikiPagesController < ApplicationController
title = params[:title]&.strip
body = params[:body].to_s
return head :unprocessable_entity if title.blank? || body.blank?
return render_unprocessable_entity('タイトルは必須です.', field: :title) if title.blank?
return render_unprocessable_entity('本文は必須です.', field: :body) if body.blank?
page = WikiPage.find(params[:id])
base_revision_id = page.current_revision.id
base_revision_id = params[:base_revision_id].presence
if params[:title].present? && params[:title].strip != page.title
return head :unprocessable_entity
ApplicationRecord.transaction do
page.lock!
old_title = page.title
tag = Tag.find_by(tag_name_id: page.tag_name_id)
if tag && title != old_title
TagVersioning.ensure_snapshot!(tag, created_by_user: current_user)
end
page.tag_name.update!(name: title) if title != old_title
message = params[:message].presence
Wiki::Commit.content!(page:,
body:,
created_user: current_user,
message:,
base_revision_id:)
if tag && title != old_title
tag.reload
TagVersionRecorder.record!(tag:, event_type: :update, created_by_user: current_user)
end
end
message = params[:message].presence
Wiki::Commit.content!(page:,
body:,
created_user: current_user,
message:,
base_revision_id:)
head :ok
end
@@ -136,7 +159,7 @@ class WikiPagesController < ApplicationController
def changes
id = params[:id].presence
q = WikiRevision.joins(wiki_page: :tag_name)
.includes(:created_user, wiki_page: :tag_name)
.includes(:created_user, wiki_page: { tag_name: :tag })
.order(id: :desc)
q = q.where(wiki_page_id: id) if id
@@ -144,7 +167,9 @@ class WikiPagesController < ApplicationController
{ revision_id: rev.id,
pred: rev.base_revision_id,
succ: nil,
wiki_page: { id: rev.wiki_page_id, title: rev.wiki_page.title },
wiki_page: { id: rev.wiki_page_id,
title: rev.wiki_page.title,
deprecated_at: rev.wiki_page.deprecated_at },
user: rev.created_user && { id: rev.created_user.id, name: rev.created_user.name },
kind: rev.kind,
message: rev.message,
+21
ファイルの表示
@@ -0,0 +1,21 @@
class GekanatorAiRun < ApplicationRecord
STATUSES = ['pending', 'running', 'succeeded', 'failed', 'blocked_budget'].freeze
belongs_to :gekanator_question_suggestion
validates :model, presence: true, length: { maximum: 255 }
validates :status, presence: true, inclusion: { in: STATUSES }
validates :input_tokens,
presence: true,
numericality: { only_integer: true, greater_than_or_equal_to: 0 }
validates :output_tokens,
presence: true,
numericality: { only_integer: true, greater_than_or_equal_to: 0 }
validates :estimated_cost_jpy,
presence: true,
numericality: { greater_than_or_equal_to: 0 }
scope :this_month, lambda {
where(created_at: Time.current.beginning_of_month..Time.current.end_of_month)
}
end
+15
ファイルの表示
@@ -0,0 +1,15 @@
class GekanatorGame < ApplicationRecord
belongs_to :user
belongs_to :guessed_post, class_name: 'Post'
belongs_to :correct_post, class_name: 'Post'
has_many :question_suggestions,
class_name: 'GekanatorQuestionSuggestion',
dependent: :delete_all
has_many :question_examples,
class_name: 'GekanatorQuestionExample',
dependent: :delete_all
validates :answers, presence: true
validates :question_count, numericality: { greater_than_or_equal_to: 0 }
validates :won, inclusion: { in: [true, false] }
end
+23
ファイルの表示
@@ -0,0 +1,23 @@
class GekanatorQuestion < ApplicationRecord
KINDS = ['tag', 'source', 'title', 'original_date', 'post_similarity'].freeze
SOURCES = ['user_suggested', 'ai_generated', 'admin_curated'].freeze
STATUSES = ['pending', 'accepted', 'rejected', 'disabled'].freeze
belongs_to :gekanator_question_suggestion, optional: true
belongs_to :created_by, class_name: 'User', optional: true
has_many :gekanator_question_examples, dependent: :delete_all
validates :kind, presence: true, inclusion: { in: KINDS }
validates :source, presence: true, inclusion: { in: SOURCES }
validates :status, presence: true, inclusion: { in: STATUSES }
validates :text, presence: true, length: { maximum: 1000 }
validates :condition, presence: true
validates :priority_weight,
presence: true,
numericality: {
greater_than: 0,
less_than_or_equal_to: 3
}
scope :accepted, -> { where(status: 'accepted') }
end
+97
ファイルの表示
@@ -0,0 +1,97 @@
class GekanatorQuestionExample < ApplicationRecord
ANSWERS = GekanatorQuestionSuggestion::ANSWERS
NON_UNKNOWN_ANSWERS = ANSWERS - ['unknown']
SOURCES = ['initial_suggestion', 'post_game_answer', 'post_game_extra'].freeze
belongs_to :gekanator_question
belongs_to :post
belongs_to :user
belongs_to :gekanator_game, optional: true
validates :answer, presence: true, inclusion: { in: ANSWERS }
validates :answer_counts, presence: true
validates :sample_count,
presence: true,
numericality: {
only_integer: true,
greater_than: 0
}
validates :source, presence: true, inclusion: { in: SOURCES }
validates :weight,
presence: true,
numericality: {
greater_than: 0
}
before_validation :normalize_learning_state
def record_answer!(answer:, source:, gekanator_game: nil)
answer = answer.to_s
raise ArgumentError, 'invalid answer' unless ANSWERS.include?(answer)
counts = normalized_answer_counts
counts[answer] += 1
self.answer_counts = counts
self.sample_count = counts.values.sum
self.gekanator_game = gekanator_game if gekanator_game.present?
self.source = source
apply_aggregated_answer!(preferred_answer: answer)
self
end
private
def normalize_learning_state
counts = normalized_answer_counts
if counts.values.sum.zero? && answer.present?
counts[answer] = 1
end
self.answer_counts = counts
self.sample_count = counts.values.sum
apply_aggregated_answer!
end
def apply_aggregated_answer!(preferred_answer: nil)
counts = normalized_answer_counts
known_counts = counts.slice(*NON_UNKNOWN_ANSWERS)
known_total = known_counts.values.sum
if known_total.zero?
self.answer = 'unknown'
self.weight = 0.1
return
else
max_count = known_counts.values.max
candidates = known_counts.select { |_answer, count| count == max_count }.keys
self.answer =
if preferred_answer.present? && candidates.include?(preferred_answer)
preferred_answer
elsif answer.present? && candidates.include?(answer)
answer
else
candidates.first
end
end
consensus = max_count.to_f / known_total
self.weight = Math.sqrt(known_total) * consensus
end
def normalized_answer_counts
base = ANSWERS.index_with(0)
answer_counts.to_h.each do |key, value|
answer_key = key.to_s
next unless ANSWERS.include?(answer_key)
base[answer_key] = value.to_i
end
base
end
end
+12
ファイルの表示
@@ -0,0 +1,12 @@
class GekanatorQuestionSuggestion < ApplicationRecord
ANSWERS = ['yes', 'no', 'partial', 'probably_no', 'unknown'].freeze
belongs_to :gekanator_game
belongs_to :user
has_many :gekanator_questions, dependent: :nullify
has_many :gekanator_ai_runs, dependent: :destroy
validates :question_text, presence: true, length: { maximum: 1000 }
validates :answer, presence: true, inclusion: { in: ANSWERS }
validates :processed, inclusion: { in: [true, false] }
end
+6 -2
ファイルの表示
@@ -1,6 +1,10 @@
class IpAddress < ApplicationRecord
validates :ip_address, presence: true, length: { maximum: 16 }
validates :banned, inclusion: { in: [true, false] }
has_many :users
has_many :user_ips, dependent: :destroy
has_many :users, through: :user_ips
def banned? = banned_at.present?
def ban! = banned? || update!(banned_at: Time.current)
def unban! = update!(banned_at: nil)
end
+80
ファイルの表示
@@ -0,0 +1,80 @@
class Material < ApplicationRecord
include MyDiscard
SOURCE_KINDS = ['uri', 'google_drive_path', 'google_drive_file',
'legacy_drive_path'].freeze
default_scope -> { kept }
belongs_to :parent, class_name: 'Material', optional: true
has_many :children, class_name: 'Material', foreign_key: :parent_id, dependent: :nullify
belongs_to :tag, optional: true
belongs_to :created_by_user, class_name: 'User', optional: true
belongs_to :updated_by_user, class_name: 'User', optional: true
has_many :material_versions, dependent: :destroy
has_many :material_export_items, dependent: :destroy
has_one_attached :file, dependent: :purge
has_one_attached :thumbnail, dependent: :purge
before_validation :assign_normalized_source_key
validates :tag_id, uniqueness: true, allow_nil: true
validates :source_kind,
inclusion: { in: SOURCE_KINDS },
allow_blank: true
validate :file_must_be_attached
validate :tag_must_be_material_category
def content_type
return nil unless file&.attached?
file.blob.content_type
end
def file_byte_size
return nil unless file&.attached?
file.blob.byte_size
end
def file_filename
return nil unless file&.attached?
file.blob.filename.to_s
end
def snapshot_export_paths
material_export_items.order(:profile).pluck(:profile, :export_path).to_h
end
private
def assign_normalized_source_key
if source_kind.blank?
self.normalized_source_key = nil
return
end
self.normalized_source_key =
MaterialSyncSuppression.normalize_source_key(source_kind:,
source_uri:,
drive_path: source_path,
drive_file_id: source_file_id)
end
def file_must_be_attached
return if url.present? || file.attached?
errors.add(:url, 'URL かファイルのどちらかは必須です.')
end
def tag_must_be_material_category
return if tag.blank? || tag.character? || tag.material?
errors.add(:tag, '素材カテゴリのタグを指定してください.')
end
end
+48
ファイルの表示
@@ -0,0 +1,48 @@
class MaterialExportItem < ApplicationRecord
VALID_PROFILES = ['legacy_drive'].freeze
belongs_to :material
belongs_to :created_by_user, class_name: 'User', optional: true
validates :profile, presence: true, inclusion: { in: VALID_PROFILES }
validates :export_path, presence: true, uniqueness: { scope: :profile }
validates :material_id, uniqueness: { scope: :profile }
validate :export_path_must_be_relative_safe_path
scope :enabled, -> { where(enabled: true) }
private
def export_path_must_be_relative_safe_path
return if export_path.blank?
if export_path.start_with?('/')
errors.add(:export_path, '絶対パスは使へません.')
end
if export_path.match?(/\A[A-Za-z]:\//)
errors.add(:export_path, '絶対パスは使へません.')
end
if export_path.include?('\\')
errors.add(:export_path, '/ 区切りで指定してください.')
end
if export_path.include?("\0")
errors.add(:export_path, 'NUL は使へません.')
end
parts = export_path.split('/')
if export_path.include?('//')
errors.add(:export_path, '空の path segment は使へません.')
end
if parts.any? { |part| part.in?(['.', '..']) }
errors.add(:export_path, '.. は使へません.')
end
if export_path.end_with?('/')
errors.add(:export_path, 'directory path は使へません.')
end
end
end
+29
ファイルの表示
@@ -0,0 +1,29 @@
class MaterialImportBlock < ApplicationRecord
MATCH_KINDS = ['sha256', 'exact_path', 'path_prefix', 'manual'].freeze
REASONS = [
'copyright_high_risk',
'copyright_takedown',
'adult_or_sensitive',
'personal_information',
'malware_or_dangerous_file',
'duplicate_or_low_quality',
'source_owner_request',
'other'
].freeze
belongs_to :created_by_user, class_name: 'User', optional: true
validates :match_kind, presence: true, inclusion: { in: MATCH_KINDS }
validates :reason, presence: true, inclusion: { in: REASONS }
validates :sha256, length: { is: 64 }, allow_blank: true
validate :match_value_must_be_present
private
def match_value_must_be_present
return if match_kind == 'manual'
return if sha256.present? || external_path_pattern.present?
errors.add(:base, 'sha256 または external_path_pattern は必須です.')
end
end
+45
ファイルの表示
@@ -0,0 +1,45 @@
class MaterialSyncSource < ApplicationRecord
SOURCE_KINDS = ['uri', 'google_drive_path', 'google_drive_file', 'legacy_drive_path'].freeze
belongs_to :created_by_user, class_name: 'User', optional: true
belongs_to :updated_by_user, class_name: 'User', optional: true
validates :name, presence: true
validates :source_kind,
presence: true,
inclusion: { in: SOURCE_KINDS }
validates :profile, presence: true, inclusion: { in: MaterialExportItem::VALID_PROFILES }
validate :source_value_must_be_present
scope :enabled, -> { where(enabled: true) }
def normalized_source_key
MaterialSyncSuppression.normalize_source_key(source_kind:,
source_uri:,
source_path:,
source_file_id:)
end
private
def source_value_must_be_present
return if source_value_present?
errors.add(:base, '同期元を指定してください.')
end
def source_value_present?
case source_kind
when 'uri'
source_uri.present?
when 'google_drive_file'
source_file_id.present? || source_uri.present?
when 'google_drive_path'
source_file_id.present? || source_uri.present?
when 'legacy_drive_path'
source_path.present?
else
normalized_source_key.present?
end
end
end
+105
ファイルの表示
@@ -0,0 +1,105 @@
class MaterialSyncSuppression < ApplicationRecord
SOURCE_KINDS = [
'uri',
'google_drive_path',
'google_drive_path_prefix',
'google_drive_file',
'legacy_drive_path',
'legacy_drive_path_prefix'
].freeze
REASONS = [
'copyright_high_risk',
'copyright_takedown',
'adult_or_sensitive',
'personal_information',
'malware_or_dangerous_file',
'duplicate_or_low_quality',
'source_owner_request',
'other'
].freeze
belongs_to :created_by_user, class_name: 'User', optional: true
before_validation :assign_normalized_source_key
validates :source_kind, presence: true, inclusion: { in: SOURCE_KINDS }
validates :reason, presence: true, inclusion: { in: REASONS }
validates :normalized_source_key, presence: true, uniqueness: true
validate :source_value_must_be_present
validate :drive_path_must_be_safe_relative_path
def self.normalize_source_key source_kind:,
source_uri: nil,
drive_path: nil,
drive_file_id: nil,
source_path: nil,
source_file_id: nil
kind = source_kind.to_s
value =
case kind
when 'uri'
source_uri.to_s.strip
when 'google_drive_path', 'google_drive_path_prefix',
'legacy_drive_path', 'legacy_drive_path_prefix'
(source_path || drive_path).to_s.strip.gsub(%r{/+}, '/')
when 'google_drive_file'
(source_file_id || drive_file_id).to_s.strip
else
''
end
return nil if kind.blank? || value.blank?
"#{ kind }:#{ value }"
end
private
def assign_normalized_source_key
normalize_path_fields!
self.normalized_source_key =
self.class.normalize_source_key(source_kind:,
source_uri:,
drive_path:,
drive_file_id:)
end
def source_value_must_be_present
return if normalized_source_key.present?
errors.add(:base, '同期元を指定してください.')
end
def drive_path_must_be_safe_relative_path
return unless source_kind.in?(
['google_drive_path', 'google_drive_path_prefix',
'legacy_drive_path', 'legacy_drive_path_prefix'])
value = drive_path.to_s
return if value.blank?
if value.start_with?('/') || value.match?(/\A[A-Za-z]:\//)
errors.add(:drive_path, '相対 path を指定してください.')
end
if value.start_with?('My Drive/', 'マイドライブ/')
errors.add(:drive_path, '同期元フォルダからの相対 path を指定してください.')
end
errors.add(:drive_path, 'NUL は使へません.') if value.include?("\0")
errors.add(:drive_path, '連続スラッシュは使へません.') if value.include?('//')
errors.add(:drive_path, '末尾スラッシュは使へません.') if value.end_with?('/')
parts = value.split('/')
if parts.any? { |part| part.in?(['.', '..']) }
errors.add(:drive_path, '. や .. は使へません.')
end
end
def normalize_path_fields!
return unless source_kind.in?(
['google_drive_path', 'google_drive_path_prefix',
'legacy_drive_path', 'legacy_drive_path_prefix'])
self.drive_path = drive_path.to_s.tr('\\', '/').strip.presence
end
end
+17
ファイルの表示
@@ -0,0 +1,17 @@
class MaterialVersion < ApplicationRecord
EVENT_TYPE_MAP = { create: 'create',
update: 'update',
discard: 'discard',
restore: 'restore' }.freeze
include VersionRecord
belongs_to :material
belongs_to :tag, optional: true
belongs_to :parent, class_name: 'Material', optional: true
belongs_to :updated_by_user, class_name: 'User', optional: true
def export_paths_hash
export_paths_json || {}
end
end
+5 -1
ファイルの表示
@@ -1,7 +1,11 @@
module MyDiscard
extend ActiveSupport::Concern
included { include Discard::Model }
included do
include Discard::Model
default_scope -> { kept }
end
class_methods do
def find_undiscard_or_create_by! attrs, &block
+7
ファイルの表示
@@ -0,0 +1,7 @@
class NicoTagVersion < ApplicationRecord
include VersionRecord
belongs_to :tag
validates :name, presence: true
end
+445 -29
ファイルの表示
@@ -1,35 +1,169 @@
class Post < ApplicationRecord
require 'date'
require 'mini_magick'
require 'nokogiri'
require 'stringio'
require 'timeout'
class RemoteThumbnailFetchFailed < StandardError; end
ORIGINAL_CREATED_INVALID_MESSAGE = 'オリジナルの作成日時の形式が不正です.'.freeze
ORIGINAL_CREATED_MINUTE_PRECISION_MESSAGE =
'オリジナルの作成日時は分単位で入力してください.'.freeze
ORIGINAL_CREATED_ORDER_MESSAGE = 'オリジナルの作成日時の順番がをかしぃです.'.freeze
ORIGINAL_CREATED_MINIMUM_RANGE_MESSAGE =
'オリジナルの作成日時の範囲は1分以上必要です.'.freeze
REMOTE_SVG_CONTENT_TYPE = 'image/svg+xml'.freeze
MAX_SVG_DIMENSION = 4_096
MAX_SVG_PIXELS = 16_777_216
THUMBNAIL_PROCESS_TIMEOUT = 5.seconds
def self.resized_thumbnail_attachment(upload, content_type: nil)
upload.rewind
bytes = upload.read
blob = Timeout.timeout(THUMBNAIL_PROCESS_TIMEOUT) do
image = image_for_thumbnail_upload(bytes, content_type:)
image.auto_orient
image.resize '180x180'
image.format 'jpg'
image.to_blob
end
{ io: StringIO.new(blob),
filename: 'resized_thumbnail.jpg',
content_type: 'image/jpeg' }
rescue Timeout::Error
raise MiniMagick::Error, 'サムネイル画像の変換に失敗しました.'
ensure
upload.rewind
end
def self.remote_thumbnail_attachment(raw_url)
response = Preview::ThumbnailFetcher.fetch_image_response(raw_url)
resized_thumbnail_attachment(
StringIO.new(response.body),
content_type: response.content_type)
rescue Preview::UrlSafety::UnsafeUrl,
Preview::ThumbnailFetcher::GenerationFailed,
Preview::HttpFetcher::FetchFailed,
Preview::HttpFetcher::FetchTimeout,
Preview::HttpFetcher::ResponseTooLarge,
Timeout::Error,
MiniMagick::Error => e
raise RemoteThumbnailFetchFailed, e.message
end
belongs_to :parent, class_name: 'Post', optional: true, foreign_key: 'parent_id'
belongs_to :uploaded_user, class_name: 'User', optional: true
has_many :post_tags, dependent: :destroy, inverse_of: :post
has_many :active_post_tags, -> { kept }, class_name: 'PostTag', inverse_of: :post
has_many :post_tags_with_discarded, -> { with_discarded }, class_name: 'PostTag'
has_many :tags, through: :active_post_tags
has_many :active_tags, -> { where(tags: { deprecated_at: nil }) },
through: :active_post_tags, source: :tag
has_many :user_post_views, dependent: :delete_all
has_many :post_similarities, dependent: :delete_all
has_many :post_versions
has_many :gekanator_guessed_games,
class_name: 'GekanatorGame',
foreign_key: :guessed_post_id,
dependent: :delete_all,
inverse_of: :guessed_post
has_many :gekanator_correct_games,
class_name: 'GekanatorGame',
foreign_key: :correct_post_id,
dependent: :delete_all,
inverse_of: :correct_post
has_many :gekanator_question_examples, dependent: :delete_all
has_many :parent_post_implications,
class_name: 'PostImplication',
foreign_key: :post_id,
dependent: :destroy,
inverse_of: :post
has_many :parents, through: :parent_post_implications, source: :parent_post
has_many :child_post_implications,
class_name: 'PostImplication',
foreign_key: :parent_post_id,
dependent: :destroy,
inverse_of: :parent_post
has_many :children, through: :child_post_implications, source: :post
has_one_attached :thumbnail
before_validation :normalise_url
attribute :version_no, :integer, default: 1
validates :url, presence: true, uniqueness: true
before_validation :normalise_url, if: :will_save_change_to_url?
validates :url, presence: true, uniqueness: true, length: { maximum: 768 }
validates :video_ms, numericality: { only_integer: true, greater_than: 0 }, allow_nil: true
validate :validate_original_created_range
validate :url_must_be_http_url
def parent_posts = parents
def child_posts = children
def sibling_posts
parent_post_ids = parent_posts.order(:id).pluck(:id)
parent_post_ids.to_h { [_1, PostImplication.where(parent_post: _1).map(&:post)] }
end
def as_json options = { }
super(options).merge({ thumbnail: thumbnail.attached? ?
Rails.application.routes.url_helpers.rails_blob_url(
thumbnail, only_path: false) :
nil })
super(options).merge(thumbnail: thumbnail.attached? ?
Rails.application.routes.url_helpers.rails_blob_url(
thumbnail, only_path: false) :
nil)
rescue
super(options).merge(thumbnail: nil)
end
def snapshot_tag_names
post_tags
.kept
.joins(tag: :tag_name)
.includes(:sections, tag: :tag_name)
.order('tag_names.name')
.map do |post_tag|
name = post_tag.tag.tag_name.name
sections = post_tag.sections.sort_by(&:begin_ms)
next name if sections.empty?
"#{ name }#{ sections.map { Post.section_literal(_1) }.join }"
end
end
def self.section_literal section
"[#{ Post.ms_to_time(section.begin_ms) }-#{ section.end_ms ? Post.ms_to_time(section.end_ms) : '' }]"
end
def self.ms_to_time ms
total_s = ms / 1_000
s = total_s % 60
min = (total_s / 60) % 60
h = total_s / 3_600
remainder_ms = ms % 1_000
base =
if h.positive?
'%d:%02d:%02d' % [h, min, s]
else
'%d:%02d' % [min, s]
end
if remainder_ms.positive?
"#{ base }.#{ remainder_ms.to_s.rjust(3, '0') }"
else
base
end
end
def snapshot_parent_post_ids = parents.order(:id).pluck(:id)
def related limit: nil
ids = post_similarities.order(cos: :desc)
ids = ids.limit(limit) if limit
@@ -44,27 +178,29 @@ class Post < ApplicationRecord
def resized_thumbnail!
return unless thumbnail.attached?
image = MiniMagick::Image.read(thumbnail.download)
image.resize '180x180'
thumbnail.purge
thumbnail.attach(io: File.open(image.path),
filename: 'resized_thumbnail.jpg',
content_type: 'image/jpeg')
thumbnail.attach(self.class.resized_thumbnail_attachment(StringIO.new(thumbnail.download)))
end
def attach_thumbnail_from_url! raw_url
thumbnail.attach(self.class.remote_thumbnail_attachment(raw_url))
end
private
def validate_original_created_range
f = original_created_from
b = original_created_before
return if f.blank? || b.blank?
return if skip_original_created_validation?
f = Time.zone.parse(f) if String === f
b = Time.zone.parse(b) if String === b
return if !(f) || !(b)
f = parse_original_created_value(:original_created_from)
b = parse_original_created_value(:original_created_before)
return if f.nil? || b.nil?
if f >= b
errors.add :original_created_before, 'オリジナルの作成日時の順番がをかしぃです.'
if b <= f
errors.add :original_created_at, ORIGINAL_CREATED_ORDER_MESSAGE
return
end
if b - f < 1.minute
errors.add :original_created_at, ORIGINAL_CREATED_MINIMUM_RANGE_MESSAGE
end
end
@@ -85,15 +221,295 @@ class Post < ApplicationRecord
def normalise_url
return if url.blank?
self.url = url.strip
self.url = PostUrlNormaliser.normalise(url) || url.strip
end
u = URI.parse(url)
return unless u in URI::HTTP
def self.image_for_thumbnail_upload(bytes, content_type: nil)
if svg_content_type?(content_type) || svg_document_bytes?(bytes)
return decode_svg_thumbnail(bytes)
end
u.host = u.host.downcase if u.host
u.path = u.path.sub(/\/\Z/, '') if u.path.present?
self.url = u.to_s
rescue URI::InvalidURIError
;
raise MiniMagick::Error, 'サムネイル画像の形式が不正です.' unless raster_thumbnail_bytes?(bytes)
decode_raster_thumbnail(bytes)
end
def self.raster_thumbnail_bytes?(bytes)
raster_thumbnail_format(bytes).present?
end
def self.remote_thumbnail_image_bytes?(bytes, content_type: nil)
return true if svg_content_type?(content_type) || svg_document_bytes?(bytes)
raster_thumbnail_bytes?(bytes)
end
def self.svg_content_type?(content_type)
content_type.to_s.split(';', 2).first.to_s.downcase.strip == REMOTE_SVG_CONTENT_TYPE
end
def self.svg_document_bytes?(bytes)
document = Nokogiri::XML(
bytes,
nil,
nil,
Nokogiri::XML::ParseOptions::STRICT |
Nokogiri::XML::ParseOptions::NONET)
document.root&.name == 'svg'
rescue Nokogiri::XML::SyntaxError
false
end
def self.decode_raster_thumbnail(bytes)
MiniMagick::Image.read(bytes)
end
def self.decode_svg_thumbnail(bytes)
MiniMagick::Image.read(sanitised_svg_bytes(bytes))
end
def self.raster_thumbnail_format(bytes)
binary = bytes.to_s.b
return 'jpeg' if binary.start_with?("\xFF\xD8\xFF".b)
return 'png' if binary.start_with?("\x89PNG\r\n\x1A\n".b)
return 'gif' if binary.start_with?('GIF87a'.b) || binary.start_with?('GIF89a'.b)
return 'webp' if binary.bytesize >= 12 &&
binary.start_with?('RIFF'.b) &&
binary.byteslice(8, 4) == 'WEBP'
nil
end
def self.sanitised_svg_bytes(bytes)
parse_options =
Nokogiri::XML::ParseOptions::STRICT |
Nokogiri::XML::ParseOptions::NONET
document = Nokogiri::XML(
bytes,
nil,
nil,
parse_options)
root = document.root
raise MiniMagick::Error, 'SVG が不正です.' if root == nil || root.name != 'svg'
raise MiniMagick::Error, 'SVG が不正です.' if document.internal_subset != nil
raise MiniMagick::Error, 'SVG が不正です.' if svg_uses_disallowed_features?(document)
width, height = svg_dimensions(root)
raise MiniMagick::Error, 'SVG が大きすぎます.' if width == nil || height == nil
if width > MAX_SVG_DIMENSION || height > MAX_SVG_DIMENSION || width * height > MAX_SVG_PIXELS
raise MiniMagick::Error, 'SVG が大きすぎます.'
end
document.to_xml
rescue Nokogiri::XML::SyntaxError
raise MiniMagick::Error, 'SVG が不正です.'
end
def self.svg_uses_disallowed_features?(document)
document.traverse.any? do |node|
next false unless node.element?
name = node.name.to_s.downcase
next true if name == 'script' || name == 'foreignobject'
next style_contains_disallowed_urls?(node.text.to_s) if name == 'style'
node.attribute_nodes.any? do |attribute|
attribute_name = attribute.name.to_s.downcase
attribute_value = attribute.value.to_s
attribute_name.start_with?('on') ||
external_svg_reference?(attribute_name, attribute_value)
end
end
end
def self.external_svg_reference?(attribute_name, attribute_value)
return external_svg_url?(attribute_value) if ['href', 'xlink:href', 'src'].include?(attribute_name)
return style_contains_disallowed_urls?(attribute_value) if attribute_name == 'style'
return svg_url_function_disallowed?(attribute_value) if attribute_value.match?(/url\s*\(/i)
false
end
def self.external_svg_url?(value)
stripped = value.to_s.strip
return false if stripped.blank? || stripped.start_with?('#')
true
end
def self.style_contains_disallowed_urls?(value)
text = value.to_s
text.match?(/@import/i) || svg_url_function_disallowed?(text)
end
def self.svg_url_function_disallowed?(value)
value.to_s.scan(/url\s*\(([^)]*)\)/i).flatten.any? do |entry|
reference = entry.to_s.strip.delete_prefix("'").delete_prefix('"').delete_suffix("'").delete_suffix('"')
reference.present? && !(reference.start_with?('#'))
end
end
def self.svg_dimensions(root)
width = svg_length_to_pixels(root['width'])
height = svg_length_to_pixels(root['height'])
return [width, height] if width && height
view_box = root['viewBox'].to_s.strip.split(/\s+/).map { Float(_1) rescue nil }
return [nil, nil] if view_box.length != 4 || view_box.any?(&:nil?)
return [nil, nil] unless view_box[2].finite? && view_box[2].positive?
return [nil, nil] unless view_box[3].finite? && view_box[3].positive?
[view_box[2], view_box[3]]
end
def self.svg_length_to_pixels(value)
return nil if value.blank?
matched = /\A([0-9]+(?:\.[0-9]+)?)(px)?\z/i.match(value.to_s.strip)
return nil if matched == nil
pixels = Float(matched[1])
return nil unless pixels.finite? && pixels.positive?
pixels
rescue ArgumentError
nil
end
private_class_method :image_for_thumbnail_upload,
:svg_content_type?,
:decode_raster_thumbnail,
:decode_svg_thumbnail,
:raster_thumbnail_format,
:sanitised_svg_bytes,
:svg_uses_disallowed_features?,
:external_svg_reference?,
:external_svg_url?,
:style_contains_disallowed_urls?,
:svg_url_function_disallowed?,
:svg_dimensions,
:svg_length_to_pixels
def parse_original_created_value field
raw_value = public_send("#{ field }_before_type_cast")
value = public_send(field)
return nil if raw_value.blank? && value.blank?
time =
case raw_value
when String
parse_original_created_string(raw_value)
when Time, ActiveSupport::TimeWithZone
raw_value.in_time_zone
else
value&.in_time_zone
end
if time.nil?
errors.add field, ORIGINAL_CREATED_INVALID_MESSAGE
return nil
end
unless minute_precision_time?(time)
errors.add field, ORIGINAL_CREATED_MINUTE_PRECISION_MESSAGE
return nil
end
time
end
def parse_original_created_string raw_value
value = raw_value.to_s.strip
return nil if value.blank?
match = value.match(/\A(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2})\z/)
if match
year = match[1].to_i
month = match[2].to_i
day = match[3].to_i
hour = match[4].to_i
minute = match[5].to_i
return nil unless valid_original_created_components?(year, month, day, hour, minute, 0)
return Time.zone.local(year, month, day, hour, minute)
end
match =
value.match(
/\A(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2})(?::(\d{2})(?:\.(\d+))?)?(Z|[+-]\d{2}:?\d{2})?\z/)
return nil if match.nil?
year = match[1].to_i
month = match[2].to_i
day = match[3].to_i
hour = match[4].to_i
minute = match[5].to_i
second = match[6]&.to_i || 0
fraction = match[7]
offset = match[8]
return nil unless valid_original_created_components?(year, month, day, hour, minute, second)
return nil if offset.present? && !(valid_original_created_offset?(offset))
if offset.present?
return Time.new(
year,
month,
day,
hour,
minute,
second + Rational(parse_original_created_nanoseconds(fraction), 1_000_000_000),
normalise_original_created_offset(offset)).in_time_zone
end
Time.zone.local(
year,
month,
day,
hour,
minute,
second).change(nsec: parse_original_created_nanoseconds(fraction))
rescue ArgumentError, TypeError
nil
end
def minute_precision_time? value
value.sec.zero? && value.nsec.zero?
end
def valid_original_created_components? year, month, day, hour, minute, second
return false unless Date.valid_date?(year, month, day)
return false unless hour.between?(0, 23)
return false unless minute.between?(0, 59)
return false unless second.between?(0, 59)
true
end
def valid_original_created_offset? value
match = value.match(/\A([+-])(\d{2}):?(\d{2})\z/)
return true if value == 'Z'
return false if match.nil?
hours = match[2].to_i
minutes = match[3].to_i
hours.between?(0, 23) && minutes.between?(0, 59)
end
def parse_original_created_nanoseconds value
return 0 if value.blank?
digits = value[0, 9].ljust(9, '0')
Integer(digits, 10)
end
def normalise_original_created_offset value
return '+00:00' if value == 'Z'
value.match?(/\A[+-]\d{2}:\d{2}\z/) ? value : "#{ value[0, 3] }:#{ value[3, 2] }"
end
def skip_original_created_validation?
return false if new_record?
return false if will_save_change_to_original_created_from?
return false if will_save_change_to_original_created_before?
true
end
end
+19
ファイルの表示
@@ -0,0 +1,19 @@
class PostImplication < ApplicationRecord
self.primary_key = :post_id, :parent_post_id
belongs_to :post, inverse_of: :parent_post_implications
belongs_to :parent_post, class_name: 'Post', inverse_of: :child_post_implications
validates :post_id, presence: true, uniqueness: { scope: :parent_post_id }
validates :parent_post_id, presence: true
validate :parent_post_mustnt_be_itself
private
def parent_post_mustnt_be_itself
if parent_post_id == post_id
errors.add :parent_post_id, '親投稿に同じ投稿を設定することはできません.'
end
end
end
+6
ファイルの表示
@@ -10,6 +10,12 @@ class PostTag < ApplicationRecord
belongs_to :created_user, class_name: 'User', optional: true
belongs_to :deleted_user, class_name: 'User', optional: true
has_many :sections, -> { order(:begin_ms) }, class_name: 'PostTagSection',
foreign_key: [:post_id, :tag_id],
primary_key: [:post_id, :tag_id],
dependent: :delete_all,
inverse_of: :post_tag
validates :post_id, presence: true
validates :tag_id, presence: true
validates :post_id, uniqueness: {
+20
ファイルの表示
@@ -0,0 +1,20 @@
class PostTagSection < ApplicationRecord
self.primary_key = :post_id, :tag_id, :begin_ms
belongs_to :post
belongs_to :tag
belongs_to :post_tag, -> { kept }, foreign_key: [:post_id, :tag_id],
primary_key: [:post_id, :tag_id],
inverse_of: :sections,
optional: true
validates :post_id, presence: true
validates :tag_id, presence: true
validates :begin_ms, presence: true,
numericality: { only_integer: true, greater_than_or_equal_to: 0 }
validates :end_ms, numericality: { only_integer: true, greater_than: :begin_ms },
allow_nil: true
end
+163
ファイルの表示
@@ -0,0 +1,163 @@
class PostUrlSanitisationRule < ApplicationRecord
include Discard::Model
class InvalidUrlError < StandardError
attr_reader :invalid_rows
def initialize(invalid_rows)
@invalid_rows = invalid_rows
ids = invalid_rows.map { _1.fetch(:post_id) }.join(', ')
super("post URL sanitisation produced invalid URLs for posts #{ ids }")
end
end
class UrlConflictError < StandardError
attr_reader :conflicts
def initialize(conflicts)
@conflicts = conflicts
urls = conflicts.map { _1.fetch(:url) }.uniq.join(', ')
super("post URL sanitisation conflicts detected for #{ urls }")
end
end
self.primary_key = :priority
default_scope -> { kept }
validates :source_pattern, presence: true, uniqueness: true
validate :source_pattern_must_be_regexp
class << self
def sanitise(url) = sanitise_with_rules(url, rules)
def apply!
rewrites = nil
Post.transaction do
compiled_rules = rules
rewrites = Post.order(:id)
.lock('FOR UPDATE')
.pluck(:id, :url)
.map do |post_id, original_url|
{ post_id:,
original_url:,
sanitised_url: sanitise_with_rules(original_url, compiled_rules) }
end
invalid_rows = rewrites.filter { invalid_sanitised_url?(_1.fetch(:sanitised_url)) }
.map { { post_id: _1.fetch(:post_id),
original_url: _1.fetch(:original_url),
sanitised_url: _1.fetch(:sanitised_url) } }
raise InvalidUrlError.new(invalid_rows) if invalid_rows.present?
conflicts = build_conflicts(rewrites)
raise UrlConflictError.new(conflicts) if conflicts.present?
changed = rewrites.filter { _1.fetch(:original_url) != _1.fetch(:sanitised_url) }
return if changed.empty?
token = SecureRandom.hex(6)
changed.each do |row|
Post.where(id: row.fetch(:post_id))
.update_all(url: temporary_url_for(row.fetch(:post_id), token))
end
changed.each do |row|
Post.where(id: row.fetch(:post_id))
.update_all(url: row.fetch(:sanitised_url))
end
end
rescue ActiveRecord::RecordNotUnique => error
conflicts = build_persisted_conflicts(rewrites)
raise error if conflicts.empty?
raise UrlConflictError.new(conflicts), cause: error
end
private
def rules = kept.order(:priority).map { |r| [Regexp.new(r.source_pattern), r.replacement] }
def sanitise_with_rules(url, compiled_rules)
compiled_rules.reduce(url.dup) do |value, (pattern, replacement)|
value.sub(pattern, replacement)
end
end
def temporary_url_for(post_id, token) =
"https://post-url-sanitising.invalid/#{ token }/#{ post_id }"
def invalid_sanitised_url?(url)
return true if url.blank?
return true if url.length > 768
parsed = URI.parse(url)
return true if !(parsed in URI::HTTP)
return true if parsed.host.blank?
false
rescue URI::InvalidURIError
true
end
def build_conflicts(rewrites)
rewrites
.group_by { _1.fetch(:sanitised_url).downcase }
.values
.filter { _1.size > 1 }
.flatten
.map { { url: _1.fetch(:sanitised_url),
post_id: _1.fetch(:post_id),
original_url: _1.fetch(:original_url) } }
end
def build_persisted_conflicts(rewrites)
return [] if rewrites.blank?
target_rows = rewrites.filter { _1.fetch(:original_url) != _1.fetch(:sanitised_url) }
target_keys = target_rows.map { _1.fetch(:sanitised_url).downcase }.uniq
return [] if target_keys.empty?
target_pairs = target_rows.to_h do |row|
[row.fetch(:post_id), row.fetch(:sanitised_url).downcase]
end
persisted_rows = Post.order(:id)
.where('LOWER(url) IN (?)', target_keys)
.pluck(:id, :url)
.reject { |post_id, original_url| target_pairs[post_id] == original_url.downcase }
.map { |post_id, original_url|
{ url: original_url,
post_id:,
original_url:,
conflict_key: original_url.downcase }
}
target_conflicts = target_rows.map { { url: _1.fetch(:sanitised_url),
post_id: _1.fetch(:post_id),
original_url: _1.fetch(:original_url),
conflict_key: _1.fetch(:sanitised_url).downcase } }
(persisted_rows + target_conflicts)
.group_by { _1.fetch(:conflict_key) }
.values
.filter { _1.size > 1 }
.flatten
.map { _1.except(:conflict_key) }
end
end
private
def source_pattern_must_be_regexp
return if source_pattern.blank?
Regexp.new(source_pattern)
rescue RegexpError
errors.add :source_pattern, '変な正規表現だね〜(笑)'
end
end
+23
ファイルの表示
@@ -0,0 +1,23 @@
class PostVersion < ApplicationRecord
include VersionRecord
belongs_to :post
belongs_to :parent, class_name: 'Post', optional: true
validates :url, presence: true
validates :video_ms, numericality: { only_integer: true, greater_than: 0 }, allow_nil: true
validate :validate_original_created_range
private
def validate_original_created_range
f = original_created_from
b = original_created_before
return if f.blank? || b.blank?
if f >= b
errors.add :original_created_before, 'オリジナルの作成日時の順番がをかしぃです.'
end
end
end
+47 -2
ファイルの表示
@@ -1,7 +1,52 @@
class Setting < ApplicationRecord
THEMES = ['system', 'light', 'dark'].freeze
# These remain in the typed settings schema to preserve existing backend
# work. The common `/users/settings` page currently surfaces only `theme`.
AUTO_FETCH_MODES = ['auto', 'manual', 'off'].freeze
WIKI_EDITOR_MODES = ['split', 'write', 'preview'].freeze
STRING_ATTRIBUTES = [
'theme',
'auto_fetch_title',
'auto_fetch_thumbnail',
'wiki_editor_mode',
].freeze
INTEGER_ATTRIBUTES = [].freeze
BOOLEAN_ATTRIBUTES = [].freeze
EDITABLE_ATTRIBUTES =
(STRING_ATTRIBUTES + INTEGER_ATTRIBUTES + BOOLEAN_ATTRIBUTES).freeze
TYPE_BY_ATTRIBUTE = {
'theme' => :string,
'auto_fetch_title' => :string,
'auto_fetch_thumbnail' => :string,
'wiki_editor_mode' => :string,
}.freeze
belongs_to :user
validates :user_id, presence: true
validates :key, presence: true, length: { maximum: 255 }
validates :value, presence: true
validates :user_id, uniqueness: true
validates :theme, inclusion: { in: THEMES }
validates :auto_fetch_title, inclusion: { in: AUTO_FETCH_MODES }
validates :auto_fetch_thumbnail, inclusion: { in: AUTO_FETCH_MODES }
validates :wiki_editor_mode, inclusion: { in: WIKI_EDITOR_MODES }
def self.defaults
{
theme: 'system',
auto_fetch_title: 'manual',
auto_fetch_thumbnail: 'manual',
wiki_editor_mode: 'split',
}
end
def self.serializable_attributes
EDITABLE_ATTRIBUTES.map(&:to_sym)
end
def serializable_hash(options = nil)
super({ only: self.class.serializable_attributes }.merge(options || { }))
end
end
+179 -24
ファイルの表示
@@ -1,3 +1,6 @@
require 'set'
class Tag < ApplicationRecord
include MyDiscard
@@ -5,7 +8,24 @@ class Tag < ApplicationRecord
;
end
default_scope -> { kept }
class DeprecatedTagNormalisationError < ArgumentError
attr_reader :tag_names
def initialize tag_names
@tag_names = Array(tag_names)
super('deprecated tags are not allowed')
end
end
class SectionLiteralParseError < ArgumentError
attr_reader :tag_name, :literal
def initialize tag_name, literal
@tag_name = tag_name
@literal = literal
super("invalid section literal for tag #{ tag_name }: #{ literal }")
end
end
has_many :post_tags, inverse_of: :tag
has_many :active_post_tags, -> { kept }, class_name: 'PostTag', inverse_of: :tag
@@ -31,10 +51,16 @@ class Tag < ApplicationRecord
class_name: 'TagSimilarity', foreign_key: :target_tag_id, dependent: :delete_all
has_many :deerjikists, dependent: :delete_all
has_many :materials
has_many :tag_versions
has_many :nico_tag_versions
belongs_to :tag_name
delegate :wiki_page, to: :tag_name
attribute :version_no, :integer, default: 1
delegate :name, to: :tag_name, allow_nil: true
validates :tag_name, presence: true
@@ -51,6 +77,7 @@ class Tag < ApplicationRecord
validate :nico_tag_name_must_start_with_nico
validate :tag_name_must_be_canonical
validate :category_must_be_deerjikist_with_deerjikists
validate :nico_tags_cannot_be_deprecated
scope :nico_tags, -> { nico }
@@ -70,44 +97,78 @@ class Tag < ApplicationRecord
(self.tag_name ||= build_tag_name).name = val
end
def deprecated? = deprecated_at?
def has_wiki = wiki_page.present?
def self.tagme
@tagme ||= find_or_create_by_tag_name!('タグ希望', category: :meta)
end
def material_id = materials.first&.id
def self.bot
@bot ||= find_or_create_by_tag_name!('bot操作', category: :meta)
end
def has_deerjikists = deerjikists.loaded? ? deerjikists.any? : deerjikists.exists?
def self.no_deerjikist
@no_deerjikist ||= find_or_create_by_tag_name!('ニジラー情報不詳', category: :meta)
end
def self.tagme = find_or_create_by_tag_name!('タグ希望', category: :meta)
def self.bot = find_or_create_by_tag_name!('bot操作', category: :meta)
def self.no_deerjikist = find_or_create_by_tag_name!('ニジラー情報不詳', category: :meta)
def self.video = find_or_create_by_tag_name!('動画', category: :meta)
def self.niconico = find_or_create_by_tag_name!('ニコニコ', category: :meta)
def self.youtube = find_or_create_by_tag_name!('YouTube', category: :meta)
def self.video
@video ||= find_or_create_by_tag_name!('動画', category: :meta)
end
def self.niconico
@niconico ||= find_or_create_by_tag_name!('ニコニコ', category: :meta)
end
def self.normalise_tags tag_names, with_tagme: true, deny_nico: true
def self.normalise_tags! tag_names, with_tagme: true,
with_no_deerjikist: true,
deny_nico: true,
deny_deprecated: false,
with_sections: false
if deny_nico && tag_names.any? { |n| n.downcase.start_with?('nico:') }
raise NicoTagNormalisationError
end
sections = { }
tags = tag_names.map do |name|
raw_name = name
pf, cat = CATEGORY_PREFIXES.find { |p, _| name.downcase.start_with?(p) } || ['', nil]
name = TagName.canonicalise(name.sub(/\A#{ pf }/i, '')).first
name = name.sub(/\A#{ pf }/i, '')
sections_by_tag = []
while (match = name.match(/\A(\S*?)\[([^\[\]\s]*)-([^\[\]\s]*)\](\S*)\z/))
name = "#{ match[1] }#{ match[4] }"
next if match[2].empty? && match[3].empty?
sections_by_tag << normalise_section_range!(
begin_raw: match[2],
end_raw: match[3],
tag_name: name)
end
if name.include?('[') || name.include?(']')
raise SectionLiteralParseError.new(raw_name, raw_name)
end
name = TagName.canonicalise(name).first
find_or_create_by_tag_name!(name, category: (cat || :general)).tap do |tag|
if deny_deprecated && tag.deprecated?
raise DeprecatedTagNormalisationError, [tag.name]
end
tag.update!(category: cat) if cat && tag.category != cat
next if sections_by_tag.empty?
sections[tag.id] ||= []
sections[tag.id].concat(sections_by_tag)
sections[tag.id] = merge_section_ranges(sections[tag.id])
sections.delete(tag.id) if sections[tag.id] == [[0, nil]]
end
end
tags << Tag.tagme if with_tagme && tags.size < 10 && tags.none?(Tag.tagme)
tags << Tag.no_deerjikist if tags.all? { |t| !(t.deerjikist?) }
tags.uniq(&:id)
tags << Tag.no_deerjikist if with_no_deerjikist && tags.all? { |t| !(t.deerjikist?) }
tags.uniq!(&:id)
if with_sections
{ tags:, sections: }
else
tags
end
end
def self.expand_parent_tags tags
@@ -133,6 +194,45 @@ class Tag < ApplicationRecord
(result + tags).uniq { |t| t.id }
end
def self.normalise_section_range! begin_raw:, end_raw:, tag_name:
begin_ms = begin_raw.empty? ? 0 : time_to_ms!(begin_raw, tag_name:)
end_ms = end_raw.empty? ? nil : time_to_ms!(end_raw, tag_name:)
if end_ms
begin_ms, end_ms = end_ms, begin_ms if begin_ms > end_ms
end_ms = begin_ms + 1 if begin_ms == end_ms
end
[begin_ms, end_ms]
end
def self.merge_section_ranges ranges
sorted_ranges = ranges.sort_by { |begin_ms, end_ms| [begin_ms, end_ms || Float::INFINITY] }
merged = []
sorted_ranges.each do |begin_ms, end_ms|
if merged.empty?
merged << [begin_ms, end_ms]
next
end
last_begin_ms, last_end_ms = merged[-1]
if last_end_ms.nil? || begin_ms <= last_end_ms
merged[-1] = [last_begin_ms, merge_section_end(last_end_ms, end_ms)]
else
merged << [begin_ms, end_ms]
end
end
merged
end
def self.merge_section_end left_end_ms, right_end_ms
return nil if left_end_ms.nil? || right_end_ms.nil?
[left_end_ms, right_end_ms].max
end
def self.find_or_create_by_tag_name! name, category:
tn = TagName.find_undiscard_or_create_by!(name: name.to_s.strip)
tn = tn.canonical if tn.canonical_id?
@@ -144,18 +244,25 @@ class Tag < ApplicationRecord
retry
end
def self.merge_tags! target_tag, source_tags
def self.merge_tags! target_tag, source_tags, created_by_user: nil
target_tag => Tag
affected_post_ids = Set.new
Tag.transaction do
TagVersioning.ensure_snapshot!(target_tag, created_by_user:)
Array(source_tags).compact.uniq.each do |source_tag|
source_tag => Tag
next if source_tag == target_tag
TagVersioning.ensure_snapshot!(source_tag, created_by_user:)
source_tag.post_tags.kept.find_each do |source_pt|
post_id = source_pt.post_id
source_pt.discard_by!(nil)
affected_post_ids << post_id
source_pt.discard_by!(created_by_user)
unless PostTag.kept.exists?(post_id:, tag: target_tag)
PostTag.create!(post_id:, tag: target_tag)
end
@@ -167,6 +274,7 @@ class Tag < ApplicationRecord
raise ActiveRecord::RecordInvalid.new(source_tag_name)
end
TagVersioning.record!(source_tag, event_type: :discard, created_by_user:)
source_tag.discard!
if source_tag.nico?
@@ -175,6 +283,13 @@ class Tag < ApplicationRecord
source_tag_name.update_columns(canonical_id: target_tag.tag_name_id,
updated_at: Time.current)
end
TagVersioning.record!(target_tag, event_type: :update, created_by_user:)
end
Post.where(id: affected_post_ids.to_a).find_each do |post|
PostVersionRecorder.ensure_snapshot!(post, created_by_user:)
PostVersionRecorder.record!(post:, event_type: :update, created_by_user:)
end
# 投稿件数を再集計
@@ -184,6 +299,14 @@ class Tag < ApplicationRecord
target_tag.reload
end
def snapshot_aliases = tag_name.aliases.kept.order(:name).pluck(:name)
def snapshot_parent_tag_ids = parents.order(:id).pluck(:id)
def snapshot_linked_tag_names
linked_tags.joins(:tag_name).order('tag_names.name').pluck('tag_names.name')
end
private
def nico_tag_name_must_start_with_nico
@@ -205,4 +328,36 @@ class Tag < ApplicationRecord
errors.add :category, 'ニジラーと紐づいてゐるタグはニジラー・カテゴリである必要があります.'
end
end
def self.time_to_ms! str, tag_name:
match =
case str
when /\A(?<seconds>\d+)(?:\.(?<ms>\d{1,3}))?\z/
{ hours: nil, minutes: nil, seconds: Regexp.last_match[:seconds],
ms: Regexp.last_match[:ms] }
when /\A(?<minutes>\d+):(?<seconds>[0-5]?\d)(?:\.(?<ms>\d{1,3}))?\z/
{ hours: nil, minutes: Regexp.last_match[:minutes],
seconds: Regexp.last_match[:seconds],
ms: Regexp.last_match[:ms] }
when /\A(?<hours>\d+):(?<minutes>[0-5]?\d):(?<seconds>[0-5]?\d)(?:\.(?<ms>\d{1,3}))?\z/
{ hours: Regexp.last_match[:hours],
minutes: Regexp.last_match[:minutes],
seconds: Regexp.last_match[:seconds],
ms: Regexp.last_match[:ms] }
end
raise SectionLiteralParseError.new(tag_name, str) unless match
total_s = match[:seconds].to_i
total_s += match[:minutes].to_i * 60 if match[:minutes]
total_s += match[:hours].to_i * 3_600 if match[:hours]
total_s * 1_000 + match[:ms].to_s.ljust(3, '0')[0, 3].to_i
end
def nico_tags_cannot_be_deprecated
if nico? && deprecated_at.present?
errors.add :deprecated_at, 'ニコタグは廃止できません.'
end
end
end
+24
ファイルの表示
@@ -6,6 +6,7 @@ class TagImplication < ApplicationRecord
validates :parent_tag_id, presence: true
validate :parent_tag_mustnt_be_itself
validate :parent_tag_mustnt_create_cycle
private
@@ -14,4 +15,27 @@ class TagImplication < ApplicationRecord
errors.add :parent_tag_id, '親タグは子タグと同一であってはなりません.'
end
end
def parent_tag_mustnt_create_cycle
return if tag_id.blank? || parent_tag_id.blank?
return if errors[:parent_tag_id].present?
seen = { }
stack = [parent_tag_id]
until stack.empty?
current_id = stack.pop
next if seen[current_id]
seen[current_id] = true
if current_id == tag_id
errors.add :parent_tag_id, '親タグに子孫タグを指定すると循環します.'
errors.add :base, 'タグの親子関係が循環します.'
return
end
stack.concat(TagImplication.where(tag_id: current_id).pluck(:parent_tag_id))
end
end
end
-2
ファイルの表示
@@ -1,8 +1,6 @@
class TagName < ApplicationRecord
include MyDiscard
default_scope -> { kept }
has_one :tag
has_one :wiki_page
+15
ファイルの表示
@@ -0,0 +1,15 @@
class TagVersion < ApplicationRecord
include VersionRecord
belongs_to :tag
enum :category, { deerjikist: 'deerjikist',
meme: 'meme',
character: 'character',
general: 'general',
material: 'material',
meta: 'meta' }, validate: true
validates :name, presence: true
validates :category, presence: true
end
+4
ファイルの表示
@@ -7,6 +7,10 @@ class Theatre < ApplicationRecord
class_name: 'TheatreWatchingUser', inverse_of: :theatre
has_many :watching_users, through: :active_theatre_watching_users, source: :user
has_many :programmes, class_name: 'TheatreProgramme'
has_many :skip_votes, class_name: 'TheatreSkipVote', dependent: :delete_all
has_many :skip_events, class_name: 'TheatreSkipEvent', dependent: :delete_all
belongs_to :host_user, class_name: 'User', optional: true
belongs_to :current_post, class_name: 'Post', optional: true
belongs_to :created_by_user, class_name: 'User'
+2 -2
ファイルの表示
@@ -1,8 +1,8 @@
class TheatreComment < ApplicationRecord
include MyDiscard
include Discard::Model
self.primary_key = :theatre_id, :no
belongs_to :theatre
belongs_to :user
end
+6
ファイルの表示
@@ -0,0 +1,6 @@
class TheatreProgramme < ApplicationRecord
self.primary_key = :theatre_id, :position
belongs_to :theatre
belongs_to :post
end
+10
ファイルの表示
@@ -0,0 +1,10 @@
class TheatreSkipEvent < ApplicationRecord
belongs_to :theatre
belongs_to :post
belongs_to :skipped_by_user, class_name: 'User'
has_many :voters, class_name: 'TheatreSkipEventVoter', dependent: :delete_all
has_many :event_tags, class_name: 'TheatreSkipEventTag', dependent: :delete_all
has_many :users, through: :voters
has_many :tags, through: :event_tags
end
+6
ファイルの表示
@@ -0,0 +1,6 @@
class TheatreSkipEventTag < ApplicationRecord
self.primary_key = :theatre_skip_event_id, :tag_id
belongs_to :theatre_skip_event
belongs_to :tag
end
+6
ファイルの表示
@@ -0,0 +1,6 @@
class TheatreSkipEventVoter < ApplicationRecord
self.primary_key = :theatre_skip_event_id, :user_id
belongs_to :theatre_skip_event
belongs_to :user
end
+7
ファイルの表示
@@ -0,0 +1,7 @@
class TheatreSkipVote < ApplicationRecord
self.primary_key = :theatre_id, :post_id, :user_id
belongs_to :theatre
belongs_to :post
belongs_to :user
end
+7 -2
ファイルの表示
@@ -4,11 +4,11 @@ class User < ApplicationRecord
validates :name, length: { maximum: 255 }
validates :inheritance_code, presence: true, length: { maximum: 64 }
validates :role, presence: true, inclusion: { in: roles.keys }
validates :banned, inclusion: { in: [true, false] }
has_many :created_posts,
class_name: 'Post', foreign_key: :uploaded_user_id, dependent: :nullify
has_many :settings
has_one :setting, dependent: :destroy
has_many :theme_slots, class_name: 'UserThemeSlot', dependent: :destroy
has_many :user_ips, dependent: :destroy
has_many :ip_addresses, through: :user_ips
has_many :user_post_views, dependent: :destroy
@@ -19,5 +19,10 @@ class User < ApplicationRecord
class_name: 'WikiPage', foreign_key: :updated_user_id, dependent: :nullify
def viewed?(post) = user_post_views.exists?(post_id: post.id)
def gte_member? = member? || admin?
def banned? = banned_at.present?
def ban! = banned? || update!(banned_at: Time.current)
def unban! = update!(banned_at: nil)
end
+27
ファイルの表示
@@ -0,0 +1,27 @@
class UserThemeSlot < ApplicationRecord
BASE_THEMES = ['light', 'dark'].freeze
SLOT_NOS = [1, 2, 3].freeze
belongs_to :user
validates :user_id, presence: true
validates :base_theme, presence: true, inclusion: { in: BASE_THEMES }
validates :slot_no, presence: true, inclusion: { in: SLOT_NOS }
validates :tokens, presence: true
validates :user_id, uniqueness: { scope: [:base_theme, :slot_no] }
validate :tokens_must_be_object
def serializable_hash(options = nil)
hash = { only: [:base_theme, :slot_no, :tokens, :created_at, :updated_at] }
super(hash.merge(options || {}))
end
private
def tokens_must_be_object
return if tokens.is_a?(Hash)
errors.add(:tokens, 'は JSON object で指定してください.')
end
end
+27
ファイルの表示
@@ -0,0 +1,27 @@
module VersionRecord
extend ActiveSupport::Concern
DEFAULT_EVENT_TYPE_MAP = { create: 'create',
update: 'update',
discard: 'discard',
restore: 'restore' }.freeze
def readonly? = persisted?
included do
event_type_map = if const_defined?(:EVENT_TYPE_MAP, false)
const_get(:EVENT_TYPE_MAP)
else
DEFAULT_EVENT_TYPE_MAP
end
belongs_to :created_by_user, class_name: 'User', optional: true
enum :event_type, event_type_map, prefix: true, validate: true
validates :version_no, presence: true, numericality: { only_integer: true, greater_than: 0 }
validates :event_type, presence: true
scope :chronological, -> { order(:version_no, :id) }
end
end
+6 -7
ファイルの表示
@@ -4,8 +4,6 @@ require 'set'
class WikiPage < ApplicationRecord
include MyDiscard
default_scope -> { kept }
has_many :wiki_revisions, dependent: :destroy
belongs_to :created_user, class_name: 'User'
belongs_to :updated_user, class_name: 'User'
@@ -15,10 +13,16 @@ class WikiPage < ApplicationRecord
foreign_key: :redirect_page_id,
dependent: :nullify
has_many :wiki_versions
attribute :version_no, :integer, default: 1
belongs_to :tag_name
validates :tag_name, presence: true
validates :body, presence: true
def title = tag_name.name
def deprecated_at = tag_name.tag&.deprecated_at
def title= val
(self.tag_name ||= build_tag_name).name = val
@@ -26,11 +30,6 @@ class WikiPage < ApplicationRecord
def current_revision = wiki_revisions.order(id: :desc).first
def body
rev = current_revision
rev.body if rev&.content?
end
def resolve_redirect limit: 10
page = self
visited = Set.new
+8
ファイルの表示
@@ -0,0 +1,8 @@
class WikiVersion < ApplicationRecord
include VersionRecord
belongs_to :wiki_page
validates :title, presence: true
validates :body, presence: true
end
+109
ファイルの表示
@@ -0,0 +1,109 @@
# frozen_string_literal: true
module MaterialRepr
BASE = { only: [:id, :url, :version_no, :source_kind, :source_uri,
:source_path, :source_file_id, :normalized_source_key,
:created_at, :updated_at],
methods: [:content_type],
include: { tag: TagRepr::BASE,
created_by_user: UserRepr::BASE,
updated_by_user: UserRepr::BASE } }.freeze
module_function
def base material, host:
material.as_json(BASE).merge(
file: if material.file.attached?
Rails.application.routes.url_helpers.rails_storage_proxy_url(
material.file, host:)
end,
thumbnail: thumbnail_url(material, host:),
thumbnail_fallback_text: thumbnail_fallback_text(material),
thumbnail_fallback_kind: thumbnail_fallback_kind(material),
media_kind: media_kind(material),
file_byte_size: material.file_byte_size,
export_paths: export_paths(material),
export_items: export_items(material))
end
def many materials, host:
materials.map { |m| base(m, host:) }
end
def list material, host:
{ id: material.id,
version_no: material.version_no,
url: material.url,
source_kind: material.source_kind,
source_uri: material.source_uri,
source_path: material.source_path,
source_file_id: material.source_file_id,
normalized_source_key: material.normalized_source_key,
tag: compact_tag(material.tag),
thumbnail: thumbnail_url(material, host:),
thumbnail_fallback_text: thumbnail_fallback_text(material),
thumbnail_fallback_kind: thumbnail_fallback_kind(material),
media_kind: media_kind(material),
content_type: material.content_type,
file_byte_size: material.file_byte_size,
created_at: material.created_at,
updated_at: material.updated_at,
export_paths: export_paths(material),
export_items: export_items(material) }
end
def list_many materials, host:
materials.map { |m| list(m, host:) }
end
def export_paths material
material.material_export_items.each_with_object({ }) do |item, hash|
hash[item.profile] = item.enabled ? item.export_path : ''
end
end
def export_items material
material.material_export_items.map do |item|
{ id: item.id,
profile: item.profile,
export_path: item.export_path,
enabled: item.enabled }
end
end
def thumbnail_url material, host:
return nil unless material.thumbnail.attached?
Rails.application.routes.url_helpers.rails_storage_proxy_url(
material.thumbnail, host:)
end
def thumbnail_fallback_text material
material.tag&.name || material.created_at&.strftime('%Y-%m-%d')
end
def thumbnail_fallback_kind material
material.tag.present? ? 'tag_name' : 'created_at'
end
def media_kind material
return 'url_only' unless material.file.attached?
content_type = material.file.blob.content_type.to_s
return 'image' if content_type.start_with?('image/')
return 'video' if content_type.start_with?('video/')
return 'audio' if content_type.start_with?('audio/')
'file_other'
end
def compact_tag tag
return nil unless tag
{ id: tag.id,
name: tag.name,
category: tag.category,
deprecated_at: tag.deprecated_at }
end
end
+19
ファイルの表示
@@ -0,0 +1,19 @@
# frozen_string_literal: true
module PostCompactRepr
module_function
def base post
return nil if post.nil?
PostRepr
.common(post)
.slice(
'id',
'title',
'url',
'thumbnail',
'thumbnail_base')
end
end
+65 -5
ファイルの表示
@@ -2,19 +2,79 @@
module PostRepr
BASE = { include: { tags: TagRepr::BASE, uploaded_user: UserRepr::BASE } }.freeze
BASE_FIELDS = [
:id,
:version_no,
:url,
:title,
:thumbnail_base,
:video_ms,
:original_created_from,
:original_created_before,
:created_at,
:updated_at
].freeze
module_function
def base post, current_user = nil
json = post.as_json(BASE)
return json.merge(viewed: false) unless current_user
json = common(post)
json['tags'] = tag_json(post)
json['uploaded_user'] = post.uploaded_user && UserRepr.base(post.uploaded_user)
json['viewed'] = current_user ? current_user.viewed?(post) : false
json
end
viewed = current_user.viewed?(post)
json.merge(viewed:)
def detail post, current_user = nil, parent_posts: [], child_posts: [],
sibling_posts: { }, related: []
base(post, current_user).merge(
'parent_posts' => cards(parent_posts),
'child_posts' => cards(child_posts),
'sibling_posts' => sibling_posts.transform_keys(&:to_s).transform_values { |posts|
cards(posts)
},
'related' => cards(related))
end
def card post
common(post).merge('parent_posts' => [], 'child_posts' => [])
end
def cards posts
posts.map { |post| card(post) }
end
def many posts, current_user = nil
posts.map { |p| base(p, current_user) }
end
def common post
BASE_FIELDS.to_h { |field| [field.to_s, post.public_send(field)] }
.merge('thumbnail' => thumbnail_url(post))
end
def tag_json post
post
.active_post_tags
.reject { _1.tag.deprecated? }
.sort_by { _1.tag.name }
.map { |post_tag|
TagRepr.inline(post_tag.tag).merge(
'children' => [],
'sections' => post_tag.sections.as_json(only: [:begin_ms, :end_ms]))
}
end
def thumbnail_url post
return nil unless post.thumbnail.attached?
Rails.application.routes.url_helpers.rails_blob_url(post.thumbnail, only_path: false)
rescue ActionController::UrlGenerationError, ArgumentError, URI::InvalidURIError => e
Rails.logger.warn(
"PostRepr.thumbnail_url failed post_id=#{post.id} " \
"attachment_id=#{post.thumbnail.attachment&.id} " \
"blob_id=#{post.thumbnail.blob&.id} " \
"error_class=#{e.class} message=#{e.message}")
nil
end
end
+8 -5
ファイルの表示
@@ -2,16 +2,19 @@
module TagRepr
BASE = { only: [:id, :category, :post_count, :created_at, :updated_at],
methods: [:name, :has_wiki] }.freeze
BASE = { only: [:id, :category, :post_count, :created_at, :updated_at, :deprecated_at],
methods: [:name, :has_wiki, :material_id, :has_deerjikists] }.freeze
module_function
def base tag
tag.as_json(BASE)
tag.as_json(BASE).merge(aliases: tag.snapshot_aliases,
parents: tag.parents.map { _1.as_json(BASE) })
end
def many tags
tags.map { |t| base(t) }
def inline tag
tag.as_json(BASE).merge(aliases: [], parents: [])
end
def many(tags) = tags.map { |t| base(t) }
end
+1 -1
ファイルの表示
@@ -2,7 +2,7 @@
module WikiPageRepr
BASE = { methods: [:title] }.freeze
BASE = { methods: [:title, :deprecated_at] }.freeze
module_function
+22
ファイルの表示
@@ -0,0 +1,22 @@
module Gekanator
class AiRunBudget
MONTHLY_LIMIT_JPY = BigDecimal('450').freeze
MAX_RUN_ESTIMATED_COST_JPY = BigDecimal('5').freeze
def self.remaining_monthly_budget_jpy
MONTHLY_LIMIT_JPY - monthly_cost_jpy
end
def self.monthly_cost_jpy
GekanatorAiRun.this_month.sum(:estimated_cost_jpy)
end
def self.exceeded?
monthly_cost_jpy >= MONTHLY_LIMIT_JPY
end
def self.exceeded_after_next_run?
monthly_cost_jpy + MAX_RUN_ESTIMATED_COST_JPY >= MONTHLY_LIMIT_JPY
end
end
end
+168
ファイルの表示
@@ -0,0 +1,168 @@
module Gekanator
class QuestionSuggestionAiConverter
# Temporary heuristic converter for #361.
# This creates pending ai_generated questions without external LLM calls;
# accepted questions are still distributed only after admin approval.
TITLE_LENGTH_RE = /\Aタイトルは\s*(\d+)\s*文字以上[??]\z/
ORIGINAL_YEAR_RE = /\Aオリジナルの投稿年は\s*(\d{4})\s*年[??]\z/
ORIGINAL_MONTH_RE = /\Aオリジナルの投稿月は\s*(\d{1,2})\s*月[??]\z/
ORIGINAL_MONTH_DAY_RE = /\Aオリジナルの投稿日は\s*(\d{1,2})\s*月\s*(\d{1,2})\s*日[??]\z/
TITLE_CONTAINS_RE = /\A題名に「(.+?)」が含まれる[??]\z/
SOURCE_RE = /\A(.+?)\s+の投稿を思[ひい]浮かべて[ゐい]る[??]\z/
def self.call(...) = new(...).call
def initialize suggestion:, user:
@suggestion = suggestion
@user = user
end
def call
suggestion.with_lock do
existing = existing_generated_question
return existing if existing
run = suggestion.gekanator_ai_runs.create!(
model: 'heuristic_converter_v1',
status: 'running',
input_tokens: 0,
output_tokens: 0,
estimated_cost_jpy: 0)
question_attributes = build_question
question =
question_attributes &&
GekanatorQuestion.create!(
**question_attributes,
source: 'ai_generated',
status: 'pending',
gekanator_question_suggestion: suggestion,
created_by: user)
run.update!(status: question ? 'succeeded' : 'failed')
question
end
rescue => error
run&.update!(status: 'failed') if run&.persisted? && run.status != 'failed'
raise error
end
private
attr_reader :suggestion, :user
def existing_generated_question
suggestion
.gekanator_questions
.where(source: 'ai_generated')
.order(id: :desc)
.first
end
def build_question
text = normalized_text
return nil if text.blank?
structured_question_for(text) || post_similarity_question_for(text)
end
def normalized_text
suggestion.question_text.to_s.gsub(/[[:space:]]+/, ' ').strip
end
def structured_question_for text
case text
when TITLE_LENGTH_RE
length = Regexp.last_match(1).to_i
return nil if length <= 0
{
text:,
kind: 'title',
condition: {
type: 'title-length-at-least',
length:
},
priority_weight: 0.95
}
when /\A題名に英数字が混じって[ゐい]る[??]\z/
{
text: '題名に英数字が混じってゐる?',
kind: 'title',
condition: { type: 'title-has-ascii' },
priority_weight: 0.95
}
when ORIGINAL_YEAR_RE
year = Regexp.last_match(1).to_i
{
text:,
kind: 'original_date',
condition: { type: 'original-year', year: },
priority_weight: 0.95
}
when ORIGINAL_MONTH_RE
month = Regexp.last_match(1).to_i
return nil unless month.between?(1, 12)
{
text:,
kind: 'original_date',
condition: { type: 'original-month', month: },
priority_weight: 0.95
}
when ORIGINAL_MONTH_DAY_RE
month = Regexp.last_match(1).to_i
day = Regexp.last_match(2).to_i
return nil unless month.between?(1, 12) && day.between?(1, 31)
{
text:,
kind: 'original_date',
condition: {
type: 'original-month-day',
monthDay: "#{ month }-#{ day }"
},
priority_weight: 0.95
}
when TITLE_CONTAINS_RE
title_text = Regexp.last_match(1).to_s.strip
return nil if title_text.blank?
{
text: "題名に「#{ title_text }」が含まれる?",
kind: 'title',
condition: { type: 'title-contains', text: title_text },
priority_weight: 0.95
}
when SOURCE_RE
host = Regexp.last_match(1).to_s.strip
return nil if host.blank?
{
text:,
kind: 'source',
condition: { type: 'source', host: },
priority_weight: 0.95
}
else
nil
end
end
def post_similarity_question_for text
return nil if suggestion.answer == 'unknown'
{
text:,
kind: 'post_similarity',
condition: {
type: 'post-similarity',
postId: suggestion.gekanator_game.correct_post_id,
answer: suggestion.answer,
threshold: 0.65
},
priority_weight: 1.0
}
end
end
end
+52
ファイルの表示
@@ -0,0 +1,52 @@
module Gekanator
class QuestionSuggestionPromoter
def self.call(...) = new(...).call
def initialize suggestion:, user:
@suggestion = suggestion
@user = user
end
def call
suggestion.with_lock do
return promoted_question if suggestion.processed?
return suggestion if suggestion.answer == 'unknown'
question = GekanatorQuestion.create!(
text: suggestion.question_text,
kind: 'post_similarity',
source: 'user_suggested',
status: 'accepted',
priority_weight: 1.2,
condition: {
type: 'post-similarity',
postId: suggestion.gekanator_game.correct_post_id,
answer: suggestion.answer,
threshold: 0.65
},
gekanator_question_suggestion: suggestion,
created_by: user)
example =
GekanatorQuestionExample.new(
gekanator_question: question,
post: suggestion.gekanator_game.correct_post,
user: user)
example.record_answer!(
answer: suggestion.answer,
source: 'initial_suggestion',
gekanator_game: suggestion.gekanator_game)
example.save!
suggestion.update!(processed: true)
question
end
end
private
attr_reader :suggestion, :user
def promoted_question
suggestion.gekanator_questions.order(id: :desc).first
end
end
end
+233
ファイルの表示
@@ -0,0 +1,233 @@
require 'json'
require 'jwt'
require 'net/http'
require 'openssl'
require 'tempfile'
require 'uri'
module GoogleDrive
class ApiClient
DRIVE_ENDPOINT = 'https://www.googleapis.com/drive/v3'
TOKEN_ENDPOINT = 'https://oauth2.googleapis.com/token'
FOLDER_MIME_TYPE = 'application/vnd.google-apps.folder'
NATIVE_FILE_MIME_TYPE_PREFIX = 'application/vnd.google-apps.'
DRIVE_SCOPE = 'https://www.googleapis.com/auth/drive.readonly'
def initialize service_account_email: ENV['GOOGLE_DRIVE_SERVICE_ACCOUNT_EMAIL'],
private_key: ENV['GOOGLE_DRIVE_PRIVATE_KEY'],
private_key_path: ENV['GOOGLE_DRIVE_PRIVATE_KEY_PATH'],
subject: ENV['GOOGLE_DRIVE_SUBJECT']
@service_account_email = service_account_email.to_s
@private_key = load_private_key(private_key, private_key_path)
@subject = subject.to_s.presence
@access_token = nil
end
def list_material_files_under_folder folder_id
files = []
each_material_file_under_folder(folder_id) { |entry| files << entry }
files
end
def each_material_file_under_folder folder_id
return enum_for(__method__, folder_id) unless block_given?
walk_folder(folder_id, nil) do |entry, relative_path|
next if entry['mimeType'] == FOLDER_MIME_TYPE
next if native_file?(entry['mimeType'])
yield build_file_entry(entry, relative_path)
end
end
def fetch_material_file file_id
metadata = get_file(file_id)
return nil if native_file?(metadata['mimeType'])
build_file_entry(metadata, metadata['name'])
end
def download_to_tempfile file_id, filename:
tempfile = Tempfile.new(['material-sync', File.extname(filename.to_s)])
tempfile.binmode
request_binary("/files/#{ file_id }",
{ alt: 'media', supportsAllDrives: true }) do |chunk|
tempfile.write(chunk.b)
end
tempfile.rewind
tempfile
rescue StandardError
tempfile&.close!
raise
end
def extract_file_id value
raw = value.to_s.strip
return nil if raw.blank?
return raw unless raw.include?('/')
uri = URI.parse(raw)
return uri.query.to_s[%r{(?:^|&)id=([^&]+)}, 1] if uri.query.present?
return uri.path[%r{/folders/([^/]+)}, 1] if uri.path.include?('/folders/')
uri.path[%r{/d/([^/]+)}, 1]
rescue URI::InvalidURIError
nil
end
private
def walk_folder folder_id, prefix, &block
list_children(folder_id).each do |entry|
relative_path = MaterialSyncExportPath.build(prefix:,
relative_path: entry['name'])
if entry['mimeType'] == FOLDER_MIME_TYPE
walk_folder(entry['id'], relative_path, &block)
next
end
block.call(entry, relative_path)
end
end
def build_file_entry entry, relative_path
{ id: entry['id'],
name: entry['name'],
mime_type: entry['mimeType'],
relative_path: MaterialSyncExportPath.normalize_path(relative_path),
sha256_checksum: entry['sha256Checksum'],
web_view_link: entry['webViewLink'],
web_content_link: entry['webContentLink'] }
end
def list_children folder_id
files = []
page_token = nil
loop do
response =
request_json('/files', {
q: "'#{ folder_id }' in parents and trashed = false",
fields: 'nextPageToken,files(id,name,mimeType,sha256Checksum,' \
'webViewLink,webContentLink)',
orderBy: 'folder,name',
pageSize: 1000,
supportsAllDrives: true,
includeItemsFromAllDrives: true,
pageToken: page_token }.compact)
files.concat(response.fetch('files'))
page_token = response['nextPageToken']
break if page_token.blank?
end
files
end
def get_file file_id
request_json("/files/#{ file_id }",
fields: 'id,name,mimeType,sha256Checksum,webViewLink,webContentLink',
supportsAllDrives: true)
end
def request_json path, params = {}
response = request(:get, path, params:)
unless response.is_a?(Net::HTTPSuccess)
raise "Google Drive API error: #{ response.code } #{ response.body }"
end
JSON.parse(response.body)
end
def request_binary path, params = {}
request(:get, path, params:) do |response|
unless response.is_a?(Net::HTTPSuccess)
raise "Google Drive download error: #{ response.code } #{ response.body }"
end
response.read_body do |chunk|
yield chunk.b
end
end
end
def request method, path, params: nil
uri = URI(DRIVE_ENDPOINT + path)
uri.query = URI.encode_www_form(params) if params.present?
klass =
case method
when :get
Net::HTTP::Get
else
raise ArgumentError, "Unsupported Google Drive request method: #{ method }"
end
request = klass.new(uri)
request['Authorization'] = "Bearer #{ access_token }"
Net::HTTP.start(uri.host,
uri.port,
use_ssl: true,
open_timeout: 10,
read_timeout: 60) do |http|
if block_given?
http.request(request) do |response|
return yield response
end
else
http.request(request)
end
end
end
def access_token
@access_token ||= fetch_access_token
end
def fetch_access_token
raise 'GOOGLE_DRIVE_SERVICE_ACCOUNT_EMAIL is required' if @service_account_email.blank?
raise 'Google Drive private key is required' if @private_key.blank?
payload = { iss: @service_account_email,
scope: DRIVE_SCOPE,
aud: TOKEN_ENDPOINT,
exp: 1.hour.from_now.to_i,
iat: Time.current.to_i }
payload[:sub] = @subject if @subject.present?
assertion = JWT.encode(payload, @private_key, 'RS256')
uri = URI(TOKEN_ENDPOINT)
request = Net::HTTP::Post.new(uri)
request.set_form_data(
grant_type: 'urn:ietf:params:oauth:grant-type:jwt-bearer',
assertion:)
response =
Net::HTTP.start(uri.host,
uri.port,
use_ssl: true,
open_timeout: 10,
read_timeout: 30) do |http|
http.request(request)
end
unless response.is_a?(Net::HTTPSuccess)
raise "Google OAuth error: #{ response.code } #{ response.body }"
end
JSON.parse(response.body).fetch('access_token')
end
def load_private_key private_key, private_key_path
raw = private_key.to_s
raw = raw.gsub('\n', "\n") if raw.present?
raw = File.read(private_key_path) if raw.blank? && private_key_path.present?
return nil if raw.blank?
OpenSSL::PKey::RSA.new(raw)
end
def native_file? mime_type
mime_type.to_s.start_with?(NATIVE_FILE_MIME_TYPE_PREFIX)
end
end
end
+57
ファイルの表示
@@ -0,0 +1,57 @@
require 'digest'
require 'json'
class MaterialFileSha256
def self.blob_metadata blob
metadata = blob.metadata
return metadata if metadata.is_a?(Hash)
return { } if metadata.blank?
JSON.parse(metadata)
rescue JSON::ParserError
{ }
end
def self.metadata_sha256 blob
blob_metadata(blob)['sha256'].presence
end
def self.assign_metadata_sha256! blob, sha256
return if sha256.blank?
metadata = blob_metadata(blob)
metadata['sha256'] = sha256
blob.metadata = metadata
blob.save! if blob.changed?
end
def self.from_blob blob, allow_download: false
sha256 = metadata_sha256(blob)
return sha256 if sha256.present?
return nil unless allow_download
begin
blob.open do |file|
sha256 = Digest::SHA256.file(file.path).hexdigest
assign_metadata_sha256!(blob, sha256)
sha256
end
rescue ActiveStorage::FileNotFoundError, ArgumentError => error
Rails.logger.warn(
"MaterialFileSha256.from_blob failed for blob_id=#{blob.id}: " \
"#{error.class}: #{error.message}",
)
nil
end
end
def self.from_upload upload
tempfile = upload&.tempfile
return nil unless tempfile
tempfile.rewind
Digest::SHA256.file(tempfile.path).hexdigest.tap do
tempfile.rewind
end
end
end
+7
ファイルの表示
@@ -0,0 +1,7 @@
class MaterialImportBlockMatcher
def self.match_for_sha256 sha256
return nil if sha256.blank?
MaterialImportBlock.find_by(match_kind: 'sha256', sha256:)
end
end
+56
ファイルの表示
@@ -0,0 +1,56 @@
class MaterialSyncExportPath
class << self
def build prefix: nil, relative_path: nil, filename: nil, source_file_id: nil
path = normalize_path(relative_path)
path = fallback_filename(filename, source_file_id) if path.blank?
normalized_prefix = normalize_path(prefix)
return path if normalized_prefix.blank?
return normalized_prefix if path.blank?
[normalized_prefix, path].join('/')
end
def normalize_path path
raw = path.to_s.delete("\0").tr('\\', '/').strip
raw = raw.sub(%r{\A[A-Za-z]:/+}, '')
raw = raw.sub(%r{\A/+}, '')
return nil if raw.blank?
segments = raw.split('/').filter_map do |segment|
normalized = normalize_segment(segment)
normalized.presence
end
return nil if segments.empty?
segments.join('/')
end
def uniquify path, source_file_id
normalized = normalize_path(path)
return normalized if normalized.blank? || source_file_id.blank?
ext = File.extname(normalized)
base = ext.present? ? normalized.delete_suffix(ext) : normalized
"#{ base }--#{ source_file_id }#{ ext }"
end
private
def fallback_filename filename, source_file_id
name = normalize_path(filename)
return name if name.present?
return nil if source_file_id.blank?
source_file_id.to_s
end
def normalize_segment segment
cleaned = segment.to_s.delete("\0").tr('\\/', '_').strip
return nil if cleaned.blank? || cleaned == '.'
return '__' if cleaned == '..'
cleaned
end
end
end
+259
ファイルの表示
@@ -0,0 +1,259 @@
require 'digest'
class MaterialSyncImporter
Result = Struct.new(:material, :action, :suppressed, :suppression, keyword_init: true)
def self.import! attributes
new(attributes).import!
end
def initialize attributes
@attributes = attributes
end
def import!
source = source_attributes
key = normalized_source_key(source)
raise ArgumentError, 'normalized_source_key is required for material sync' if key.blank?
suppression = suppression_for(source)
if suppression
return Result.new(material: nil,
action: :suppressed,
suppressed: true,
suppression:)
end
material =
Material
.unscoped
.find_or_initialize_by(normalized_source_key: key)
if unchanged?(material, source)
return Result.new(material:,
action: :unchanged,
suppressed: false,
suppression: nil)
end
event_type =
if material.new_record?
:create
elsif material.discarded?
:restore
else
:update
end
uploaded_blob = uploaded_blob!
if @import_block
return Result.new(material: nil,
action: :suppressed,
suppressed: true,
suppression: @import_block)
end
Material.transaction do
if material.persisted?
MaterialVersionRecorder.ensure_snapshot!(material,
created_by_user: @attributes[:updated_by_user])
end
material.assign_attributes(material_attributes_for(material).merge(source))
material.discarded_at = nil if material.respond_to?(:discarded_at=)
material.file.attach(uploaded_blob) if uploaded_blob
material.save!
upsert_export_item!(material)
MaterialVersionRecorder.record!(material:,
event_type:,
created_by_user: @attributes[:updated_by_user])
end
MaterialThumbnailGenerator.generate!(material)
Result.new(material:,
action: event_type == :create ? :imported : :updated,
suppressed: false,
suppression: nil)
rescue StandardError
uploaded_blob&.purge_later
raise
ensure
close_tempfile!
end
private
def source_attributes
{ source_kind: @attributes[:source_kind],
source_uri: @attributes[:source_uri],
source_path: @attributes[:source_path],
source_file_id: @attributes[:source_file_id] }
end
def normalized_source_key source
MaterialSyncSuppression.normalize_source_key(source_kind: source[:source_kind],
source_uri: source[:source_uri],
source_path: source[:source_path],
source_file_id: source[:source_file_id])
end
def material_attributes_for material
attrs =
@attributes
.except(:source_kind, :source_uri, :source_path, :source_file_id,
:file_blob, :file_tempfile, :file_downloader,
:filename, :content_type,
:file_sha256, :export_path, :profile, :tag, :url)
if material.new_record?
attrs[:tag] = @attributes[:tag] if @attributes.key?(:tag)
attrs[:url] = @attributes[:url] if @attributes.key?(:url)
return attrs
end
if @attributes.key?(:tag) && @attributes[:tag].present?
attrs[:tag] = @attributes[:tag]
end
if @attributes.key?(:url) && @attributes[:url].present?
attrs[:url] = @attributes[:url]
end
attrs
end
def suppression_for source
if source[:source_kind] == 'google_drive_file' && source[:source_path].present?
return MaterialSyncSuppressionMatcher.match_google_drive_candidate(
drive_file_id: source[:source_file_id],
relative_path: source[:source_path])
end
MaterialSyncSuppressionMatcher.match(**source)
end
def uploaded_blob!
return @attributes[:file_blob] if @attributes[:file_blob]
tempfile = upload_tempfile
return nil unless tempfile
tempfile.rewind
file_sha256 = @attributes[:file_sha256] || Digest::SHA256.file(tempfile.path).hexdigest
@attributes[:file_sha256] = file_sha256
@import_block = MaterialImportBlockMatcher.match_for_sha256(file_sha256)
if @import_block
tempfile.rewind
return nil
end
blob = ActiveStorage::Blob.create_and_upload!(
io: tempfile,
filename: @attributes[:filename],
content_type: @attributes[:content_type])
MaterialFileSha256.assign_metadata_sha256!(blob, file_sha256)
tempfile.rewind
blob
end
def upload_tempfile
return @upload_tempfile if defined?(@upload_tempfile)
@upload_tempfile = @attributes[:file_tempfile]
if @upload_tempfile.blank? && @attributes[:file_downloader]
@upload_tempfile = @attributes[:file_downloader].call
end
@upload_tempfile
end
def upsert_export_item! material
export_path = resolved_export_path(material)
return if export_path.blank?
item = material.material_export_items.find_or_initialize_by(profile: effective_profile)
item.export_path = export_path
item.enabled = true
item.created_by_user ||= @attributes[:created_by_user]
item.save!
end
def export_path_taken_by_other? material, profile, export_path
MaterialExportItem
.where(profile:, export_path:)
.where.not(material_id: material.id)
.exists?
end
def close_tempfile!
tempfile = upload_tempfile
return unless tempfile
tempfile.close! unless tempfile.closed?
rescue StandardError
nil
end
def unchanged? material, source
return false if material.new_record? || material.discarded?
return false unless same_source_attributes?(material, source)
return false unless same_export_path?(material)
return false unless same_material_attributes?(material)
same_file_snapshot?(material)
end
def same_source_attributes? material, source
material.source_kind == source[:source_kind] \
&& material.source_uri == source[:source_uri] \
&& material.source_path == source[:source_path] \
&& material.source_file_id == source[:source_file_id]
end
def same_export_path? material
expected = resolved_export_path(material)
current = material.material_export_items.find { |item| item.profile == effective_profile }
return current.blank? if expected.blank?
current&.enabled && current.export_path == expected
end
def same_material_attributes? material
same_tag_attribute?(material) && same_url_attribute?(material)
end
def same_file_snapshot? material
expected_sha256 = @attributes[:file_sha256].to_s.presence
if expected_sha256.present?
return false unless material.file.attached?
return MaterialFileSha256.metadata_sha256(material.file.blob) == expected_sha256
end
!material.file.attached?
end
def effective_profile
@attributes[:profile].presence || 'legacy_drive'
end
def resolved_export_path material
export_path = @attributes[:export_path].to_s.strip
return export_path if export_path.blank?
return export_path unless export_path_taken_by_other?(material, effective_profile, export_path)
MaterialSyncExportPath.uniquify(export_path, @attributes[:source_file_id])
end
def same_tag_attribute? material
return true unless @attributes.key?(:tag)
return true if @attributes[:tag].blank?
material.tag_id == @attributes[:tag].id
end
def same_url_attribute? material
return true unless @attributes.key?(:url)
return true if @attributes[:url].blank?
material.url == @attributes[:url]
end
end
+251
ファイルの表示
@@ -0,0 +1,251 @@
class MaterialSyncRunner
Result = Struct.new(:imported, :updated, :unchanged, :suppressed, :failed,
:errors, keyword_init: true)
def self.sync_enabled!
results = MaterialSyncSource.enabled.order(:id).map { |source| new(source).sync! }
Result.new(imported: results.sum(&:imported),
updated: results.sum(&:updated),
unchanged: results.sum(&:unchanged),
suppressed: results.sum(&:suppressed),
failed: results.sum(&:failed),
errors: results.flat_map(&:errors))
end
def initialize source
@source = source
end
def sync!
result = Result.new(imported: 0, updated: 0, unchanged: 0,
suppressed: 0, failed: 0, errors: [])
if @source.source_kind == 'google_drive_path'
sync_google_drive_path!(result)
else
candidates.each do |candidate|
next if candidate.blank?
sync_candidate!(candidate, result)
end
end
@source.update!(last_synced_at: Time.current)
log_result(result)
result
rescue NotImplementedError, StandardError => e
result.failed += 1
result.errors << { source_id: @source.id, error: e.message }
log_result(result)
result
end
private
def candidates
case @source.source_kind
when 'uri'
[uri_candidate]
when 'google_drive_file'
[google_drive_file_candidate]
when 'legacy_drive_path'
raise NotImplementedError, 'legacy_drive_path material sync is not implemented'
else
raise NotImplementedError, "Unsupported material sync source_kind: #{ @source.source_kind }"
end
end
def uri_candidate
{ source_kind: @source.source_kind,
source_uri: @source.source_uri,
source_path: @source.source_path,
source_file_id: @source.source_file_id,
url: @source.source_uri,
tag: nil,
created_by_user: @source.created_by_user,
updated_by_user: @source.updated_by_user || @source.created_by_user }
end
def sync_candidate! candidate, result
block = MaterialImportBlockMatcher.match_for_sha256(candidate[:file_sha256])
if block
result.suppressed += 1
Rails.logger.info(
material_sync_log(action: 'suppressed',
reason: block.reason,
normalized_source_key: candidate_normalized_source_key(candidate)))
return
end
suppression = suppression_for(candidate)
if suppression
result.suppressed += 1
Rails.logger.info(
material_sync_log(action: 'suppressed',
normalized_source_key: suppression.normalized_source_key))
return
end
import = MaterialSyncImporter.import!(candidate)
result.public_send("#{ import.action }=", result.public_send(import.action) + 1)
rescue StandardError => e
result.failed += 1
result.errors << { source_id: @source.id,
normalized_source_key: candidate_normalized_source_key(candidate),
error: e.message }
Rails.logger.warn(
material_sync_log(action: 'failed',
error: e.message,
normalized_source_key: candidate_normalized_source_key(candidate)))
ensure
close_candidate_tempfile(candidate)
end
def google_drive_path_candidates
folder_id = google_drive_folder_id
Enumerator.new do |entries|
drive_client.each_material_file_under_folder(folder_id).each do |entry|
entries << build_google_drive_candidate(entry)
end
end
end
def sync_google_drive_path! result
folder_id = google_drive_folder_id
scanned_count = 0
drive_client.each_material_file_under_folder(folder_id) do |entry|
scanned_count += 1
sync_candidate!(build_google_drive_candidate(entry), result)
log_google_drive_progress(folder_id, scanned_count, result) if progress_log_scan_count?(scanned_count)
end
log_google_drive_progress(folder_id, scanned_count, result, summary: true)
end
def google_drive_file_candidate
entry = drive_client.fetch_material_file(google_drive_file_id)
return nil unless entry
build_google_drive_candidate(entry, single_file: true)
end
def build_google_drive_candidate entry, single_file: false
relative_path =
if single_file
nil
else
entry[:relative_path]
end
export_path =
MaterialSyncExportPath.build(prefix: @source.export_path_prefix,
relative_path:,
filename: entry[:name],
source_file_id: entry[:id])
{ source_kind: 'google_drive_file',
source_uri: entry[:web_view_link] || google_drive_file_url(entry[:id]),
source_path: relative_path.presence || MaterialSyncExportPath.normalize_path(entry[:name]),
source_file_id: entry[:id],
filename: entry[:name],
content_type: entry[:mime_type],
file_downloader: lambda {
drive_client.download_to_tempfile(entry[:id], filename: entry[:name])
},
file_sha256: entry[:sha256_checksum],
export_path:,
profile: @source.profile,
tag: nil,
url: nil,
created_by_user: @source.created_by_user,
updated_by_user: @source.updated_by_user || @source.created_by_user }
end
def candidate_normalized_source_key candidate
MaterialSyncSuppression.normalize_source_key(source_kind: candidate[:source_kind],
source_uri: candidate[:source_uri],
source_path: candidate[:source_path],
source_file_id: candidate[:source_file_id])
end
def suppression_for candidate
if candidate[:source_kind] == 'google_drive_file' && candidate[:source_path].present?
return MaterialSyncSuppressionMatcher.match_google_drive_candidate(
drive_file_id: candidate[:source_file_id],
relative_path: candidate[:source_path])
end
MaterialSyncSuppressionMatcher.match(source_kind: candidate[:source_kind],
source_uri: candidate[:source_uri],
source_path: candidate[:source_path],
source_file_id: candidate[:source_file_id])
end
def drive_client
@drive_client ||= GoogleDrive::ApiClient.new
end
def google_drive_folder_id
google_drive_target_id.tap do |id|
if id.blank?
raise ArgumentError, 'google_drive_path source_file_id or source_uri is required'
end
end
end
def google_drive_file_id
google_drive_target_id.tap do |id|
if id.blank?
raise ArgumentError, 'google_drive_file source_file_id or source_uri is required'
end
end
end
def google_drive_target_id
@source.source_file_id.presence || drive_client.extract_file_id(@source.source_uri)
end
def google_drive_file_url file_id
"https://drive.google.com/file/d/#{ file_id }/view"
end
def close_candidate_tempfile candidate
tempfile = candidate[:file_tempfile]
return unless tempfile
tempfile.close! unless tempfile.closed?
rescue StandardError
nil
end
def log_result result
Rails.logger.info(material_sync_log(imported: result.imported,
updated: result.updated,
unchanged: result.unchanged,
suppressed: result.suppressed,
failed: result.failed))
end
def progress_log_scan_count? scanned_count
scanned_count == 1 || (scanned_count % 50).zero?
end
def log_google_drive_progress folder_id, scanned_count, result, summary: false
Rails.logger.info(
material_sync_log(folder_id:,
scanned_count:,
imported: result.imported,
updated: result.updated,
unchanged: result.unchanged,
suppressed: result.suppressed,
failed: result.failed,
progress: summary ? 'summary' : 'scan'))
end
def material_sync_log fields
{ material_sync_source_id: @source.id,
material_sync_source_name: @source.name }.merge(fields).to_json
end
end
+68
ファイルの表示
@@ -0,0 +1,68 @@
class MaterialSyncSuppressionMatcher
def self.match source_kind:,
source_uri: nil,
drive_path: nil,
drive_file_id: nil,
source_path: nil,
source_file_id: nil
key = MaterialSyncSuppression.normalize_source_key(source_kind:,
source_uri:,
drive_path:,
drive_file_id:,
source_path:,
source_file_id:)
return nil if key.blank?
MaterialSyncSuppression.find_by(normalized_source_key: key)
end
def self.match_google_drive_candidate drive_file_id:, relative_path:
normalized_path = validate_relative_path!(relative_path)
exact = [
normalize_key('google_drive_file', drive_file_id),
normalize_key('google_drive_path', normalized_path),
normalize_key('legacy_drive_path', normalized_path),
].compact
suppression =
MaterialSyncSuppression.find_by(normalized_source_key: exact)
return suppression if suppression
MaterialSyncSuppression
.where(source_kind: ['google_drive_path_prefix', 'legacy_drive_path_prefix'])
.where('drive_path = :path OR :path LIKE CONCAT(drive_path, "/%")', path: normalized_path)
.order(:id)
.first
end
def self.suppressed?(...)
match(...).present?
end
def self.validate_relative_path! relative_path
raw = relative_path.to_s.delete("\0").tr('\\', '/').strip
raise ArgumentError, 'relative_path is required' if raw.blank?
raise ArgumentError, 'relative_path must be relative' if raw.start_with?('/')
raise ArgumentError, 'relative_path must be relative' if raw.match?(/\A[A-Za-z]:\//)
if raw.start_with?('My Drive/', 'マイドライブ/')
raise ArgumentError, 'relative_path must be relative to source folder'
end
raise ArgumentError, 'relative_path must not contain //' if raw.include?('//')
raise ArgumentError, 'relative_path must not end with /' if raw.end_with?('/')
parts = raw.split('/')
if parts.any? { |part| part.in?(['.', '..']) }
raise ArgumentError, 'relative_path must not contain dot segments'
end
raw
end
def self.normalize_key source_kind, value
return nil if value.blank?
MaterialSyncSuppression.normalize_source_key(source_kind:, source_path: value,
source_file_id: value)
end
end
+51
ファイルの表示
@@ -0,0 +1,51 @@
class MaterialSyncSuppressionRegistrar
def self.create! attributes, created_by_user:
new(attributes, created_by_user:).create!
end
def initialize attributes, created_by_user:
@attributes = attributes
@created_by_user = created_by_user
end
def create!
suppression = nil
MaterialSyncSuppression.transaction do
suppression = MaterialSyncSuppression.create!(
@attributes.merge(created_by_user: @created_by_user))
discard_existing_materials!(suppression)
end
suppression
end
private
def discard_existing_materials! suppression
matching_materials(suppression).find_each do |material|
MaterialVersionRecorder.ensure_snapshot!(material, created_by_user: @created_by_user)
material.discard!
MaterialVersionRecorder.record!(material:,
event_type: :discard,
created_by_user: @created_by_user)
end
end
def matching_materials suppression
materials = Material.unscoped.kept
case suppression.source_kind
when 'google_drive_path', 'legacy_drive_path'
materials.where(source_path: suppression.drive_path)
when 'google_drive_path_prefix', 'legacy_drive_path_prefix'
path = suppression.drive_path.to_s
materials.where(
'source_path = :path OR source_path LIKE :prefix',
path:,
prefix: "#{ path }/%")
else
materials.where(normalized_source_key: suppression.normalized_source_key)
end
end
end
+126
ファイルの表示
@@ -0,0 +1,126 @@
# frozen_string_literal: true
require 'mini_magick'
require 'open3'
require 'tempfile'
class MaterialThumbnailGenerator
SIZE = '180x180'
class << self
def generate! material
new(material).generate!
end
end
def initialize material
@material = material
@ffmpeg_stderr = []
end
def generate!
return log_result(:no_file) unless @material.file.attached?
return log_result(:unsupported_content_type) unless image? || video?
@material.file.blob.open do |file|
thumbnail = image? ? image_thumbnail(file.path) : video_thumbnail(file.path)
return log_result(:generation_failed) unless thumbnail
return attach_thumbnail(thumbnail)
end
rescue ActiveStorage::FileNotFoundError => e
log_result(:file_not_found, error: e)
rescue MiniMagick::Error => e
log_result(:mini_magick_error, error: e)
rescue ArgumentError, StandardError => e
log_result(:generation_failed, error: e)
end
private
def image? = content_type.start_with?('image/')
def video? = content_type.start_with?('video/')
def content_type
return nil unless @material.file.attached?
@material.file.blob.content_type.to_s
end
def image_thumbnail path
image = MiniMagick::Image.open(path)
image.resize(SIZE)
image.format('jpg')
image
end
def video_thumbnail path
[1, 0].each do |seconds|
tempfile = Tempfile.new(['material-thumbnail', '.jpg'])
tempfile.close
ok = extract_video_frame(path, tempfile.path, seconds)
next unless ok && File.size?(tempfile.path)
return image_thumbnail(tempfile.path)
ensure
tempfile&.unlink
end
nil
end
def extract_video_frame input_path, output_path, seconds
_stdout, stderr, status =
Open3.capture3('ffmpeg',
'-y',
'-ss', seconds.to_s,
'-i', input_path,
'-frames:v', '1',
'-f', 'image2',
output_path)
@ffmpeg_stderr << stderr if stderr.present?
status.success?
rescue Errno::ENOENT => e
@ffmpeg_stderr << "ffmpeg unavailable: #{ e.message }"
false
end
def attach_thumbnail image
blob = nil
File.open(image.path) do |io|
blob = ActiveStorage::Blob.create_and_upload!(
io:,
filename: 'material-thumbnail.jpg',
content_type: 'image/jpeg')
end
@material.thumbnail.attach(blob)
log_result(:attached)
rescue StandardError => e
blob&.purge_later
log_result(:attach_failed, error: e)
end
def log_result result, error: nil
log_payload = { material_id: @material.id,
file_blob_id: file_blob_id,
content_type:,
result:,
error_class: error&.class&.name,
error_message: error&.message,
ffmpeg_stderr: @ffmpeg_stderr.join("\n").presence }
if [:attached, :no_file, :unsupported_content_type].include?(result)
Rails.logger.info("Material thumbnail generation: #{ log_payload.to_json }")
else
Rails.logger.warn("Material thumbnail generation: #{ log_payload.to_json }")
end
result
end
def file_blob_id
return nil unless @material.file.attached?
@material.file.blob.id
end
end
+73
ファイルの表示
@@ -0,0 +1,73 @@
class MaterialVersionRecorder < VersionRecorder
EVENT_TYPES = ['create', 'update', 'discard', 'restore'].freeze
def self.record! material:, event_type:, created_by_user:, file_snapshot: nil
new(material:, event_type:, created_by_user:, file_snapshot:).record!
end
def initialize material:, event_type:, created_by_user:, file_snapshot: nil
@file_snapshot = file_snapshot
super(record: material, event_type:, created_by_user:)
end
def self.ensure_snapshot! material, created_by_user:
return if material.material_versions.exists?
record!(material:, event_type: :create,
created_by_user: material.created_by_user || created_by_user)
end
private
def version_class = MaterialVersion
def version_association = :material_versions
def record_key = :material
def snapshot_attributes
blob = @record.file.attached? ? @record.file.blob : nil
file_snapshot = build_file_snapshot(blob)
{ url: @record.url,
parent: @record.parent,
tag: @record.tag,
tag_name: @record.tag&.name,
tag_category: @record.tag&.category,
source_kind: @record.source_kind,
source_uri: @record.source_uri,
source_path: @record.source_path,
source_file_id: @record.source_file_id,
normalized_source_key: @record.normalized_source_key,
export_paths_json: @record.snapshot_export_paths,
discarded_at: @record.discarded_at,
file_blob_id: file_snapshot[:file_blob_id],
file_filename: file_snapshot[:file_filename],
file_content_type: file_snapshot[:file_content_type],
file_byte_size: file_snapshot[:file_byte_size],
file_checksum: file_snapshot[:file_checksum],
file_sha256: file_snapshot[:file_sha256] }
end
def build_file_snapshot blob
return @file_snapshot if @file_snapshot
return empty_file_snapshot unless blob
{ file_blob_id: blob.id,
file_filename: blob.filename.to_s,
file_content_type: blob.content_type,
file_byte_size: blob.byte_size,
file_checksum: blob.checksum,
file_sha256: MaterialFileSha256.metadata_sha256(blob) }
end
def empty_file_snapshot
{ file_blob_id: nil,
file_filename: nil,
file_content_type: nil,
file_byte_size: nil,
file_checksum: nil,
file_sha256: nil }
end
def event_types = self.class::EVENT_TYPES
end
+148
ファイルの表示
@@ -0,0 +1,148 @@
require 'stringio'
require 'zlib'
# Initial implementation keeps every file payload and the final ZIP in memory.
# Keep this service boundary stable so job/cached export paths can replace it later.
class MaterialZipExporter
Entry = Struct.new(:path, :data, :mtime, keyword_init: true)
MissingFile = Struct.new(:material_id, :export_path, :blob_id, :filename, keyword_init: true)
class EmptyExportError < StandardError; end
class DuplicatePathError < StandardError; end
class MissingFileError < StandardError
attr_reader :missing_files
def initialize missing_files
@missing_files = missing_files
super("Missing files: #{missing_files.map(&:export_path).join(', ')}")
end
end
def initialize profile: 'legacy_drive', tag_id: nil
@profile = profile.presence || 'legacy_drive'
@tag_id = tag_id.presence
end
def export
entries = build_entries
raise EmptyExportError if entries.empty?
ZipWriter.write(entries)
end
private
def build_entries
rows = MaterialExportItem
.enabled
.includes(material: { file_attachment: :blob })
.joins(:material)
.merge(Material.kept)
.where(profile: @profile)
.order(:export_path)
rows = rows.where(materials: { tag_id: @tag_id }) if @tag_id
missing_files = []
entries = rows.filter_map do |item|
material = item.material
next unless material.file.attached?
data = download_blob(item, missing_files)
next unless data
Entry.new(path: item.export_path,
data:,
mtime: material.updated_at || Time.current)
end
raise MissingFileError.new(missing_files) if missing_files.any?
paths = entries.map(&:path)
duplicated = paths.find { |path| paths.count(path) > 1 }
raise DuplicatePathError, duplicated if duplicated
entries
end
def download_blob item, missing_files
blob = item.material.file.blob
blob.download
rescue ActiveStorage::FileNotFoundError
missing_files << MissingFile.new(
material_id: item.material_id,
export_path: item.export_path,
blob_id: blob.id,
filename: blob.filename.to_s,
)
nil
end
class ZipWriter
VERSION_NEEDED = 20
GP_FLAG = 0x0800
COMPRESSION_STORE = 0
def self.write entries
new(entries).write
end
def initialize entries
@entries = entries
@central_directory = []
end
def write
io = StringIO.new(''.b)
@entries.each do |entry|
write_entry(io, entry)
end
central_start = io.pos
@central_directory.each { |header| io.write(header) }
central_size = io.pos - central_start
io.write([0x06054b50, 0, 0, @entries.size, @entries.size,
central_size, central_start, 0].pack('VvvvvVVv'))
io.string
end
private
def write_entry io, entry
path = entry.path.b
data = entry.data.b
crc32 = Zlib.crc32(data)
dos_time, dos_date = dos_timestamp(entry.mtime)
offset = io.pos
local_header = [0x04034b50, VERSION_NEEDED, GP_FLAG, COMPRESSION_STORE,
dos_time, dos_date, crc32, data.bytesize, data.bytesize,
path.bytesize, 0].pack('VvvvvvVVVvv')
io.write(local_header)
io.write(path)
io.write(data)
@central_directory << central_header(path:, crc32:, size: data.bytesize,
dos_time:, dos_date:, offset:)
end
def central_header path:, crc32:, size:, dos_time:, dos_date:, offset:
[0x02014b50, VERSION_NEEDED, VERSION_NEEDED, GP_FLAG, COMPRESSION_STORE,
dos_time, dos_date, crc32, size, size, path.bytesize, 0, 0, 0, 0, 0,
offset].pack('VvvvvvvVVVvvvvvVV') + path
end
def dos_timestamp time
local = time.to_time
dos_time = (local.hour << 11) | (local.min << 5) | (local.sec / 2)
dos_date = ((local.year - 1980) << 9) | (local.month << 5) | local.day
[dos_time, dos_date]
end
end
end
+19
ファイルの表示
@@ -0,0 +1,19 @@
class NicoTagVersionRecorder < VersionRecorder
def self.record! tag:, event_type:, created_by_user:
new(tag:, event_type:, created_by_user:).record!
end
def initialize tag:, event_type:, created_by_user:
super(record: tag, event_type:, created_by_user:)
end
private
def version_class = NicoTagVersion
def version_association = :nico_tag_versions
def record_key = :tag
def snapshot_attributes
{ name: @record.name, linked_tags: @record.snapshot_linked_tag_names.join(' ') }
end
end
+200
ファイルの表示
@@ -0,0 +1,200 @@
class PostBulkCreator
def initialize actor:, posts:, thumbnails:
@actor_id = actor.id
@posts = posts
@thumbnails = thumbnails
end
def run
results = Array.new(@posts.length)
mutex = Mutex.new
next_index = 0
workers = Array.new(2) do
Thread.new do
Rails.application.executor.wrap do
ActiveRecord::Base.connection_pool.with_connection do
actor = User.find(@actor_id)
loop do
index = nil
begin
index = mutex.synchronize do
current = next_index
next_index += 1
current
end
break if index >= @posts.length
attributes = @posts[index]
results[index] = create_row(actor, attributes, index)
rescue StandardError => e
Rails.logger.error(
"post_bulk_creator_worker_failure #{ { error: e.class.name,
message: e.message,
index: }.to_json }")
results[index] = {
status: 'failed',
recoverable: false,
errors: { base: ['登録中にエラーが発生しました.'] },
base_errors: [] }
end
end
end
end
end
end
workers.each(&:join)
results.each_index do |index|
next if results[index].present?
results[index] = {
status: 'failed',
recoverable: false,
errors: { base: ['登録中にエラーが発生しました.'] },
base_errors: [] }
end
{ results: }
end
private
def create_row actor, attributes, index
preflight =
PostCreatePreflight.new(
attributes: attributes,
thumbnail: thumbnail_for(index, attributes)).run
if preflight[:existing_post_id].present?
return {
status: 'skipped',
existing_post_id: preflight[:existing_post_id],
existing_post: preflight[:existing_post] }
end
post = PostCreator.new(
actor: actor,
attributes: normalised_attributes(attributes, preflight, index)).create!
result = {
status: 'created',
post: { id: post.id } }
result[:field_warnings] = preflight[:field_warnings] if preflight[:field_warnings].present?
result[:base_warnings] = preflight[:base_warnings] if preflight[:base_warnings].present?
result
rescue PostCreatePreflight::ValidationFailed => e
{
status: 'failed',
recoverable: true,
errors: e.fields,
base_errors: e.base_errors }
rescue ActiveRecord::RecordInvalid => e
existing_post = existing_post_for_race(attributes, e.record)
if existing_post.present?
return {
status: 'skipped',
existing_post_id: existing_post[:id],
existing_post: existing_post }
end
{
status: 'failed',
recoverable: true,
errors: e.record.errors.to_hash,
base_errors: e.record.errors[:base] }
rescue ActiveRecord::RecordNotUnique => e
if e.message.include?('index_posts_on_url')
existing_post = existing_post_for_race(attributes)
return {
status: 'skipped',
existing_post_id: existing_post[:id],
existing_post: existing_post } if existing_post.present?
end
Rails.logger.error(
"post_bulk_creator_record_not_unique #{ { error: e.class.name,
message: e.message }.to_json }")
{
status: 'failed',
recoverable: false,
errors: { base: ['登録中にエラーが発生しました.'] },
base_errors: [] }
rescue Tag::NicoTagNormalisationError
{
status: 'failed',
recoverable: true,
errors: { tags: ['ニコニコ・タグは直接指定できません.'] },
base_errors: [] }
rescue Tag::DeprecatedTagNormalisationError
{
status: 'failed',
recoverable: true,
errors: { tags: ['廃止済みタグは付与できません.'] },
base_errors: [] }
rescue PostCreator::VideoMsParseError
{
status: 'failed',
recoverable: true,
errors: { video_ms: ['動画時間の記法が不正です.'] },
base_errors: [] }
rescue Post::RemoteThumbnailFetchFailed
{
status: 'failed',
recoverable: true,
errors: { thumbnail_base: ['サムネイル画像の取得に失敗しました.'] },
base_errors: [] }
rescue ArgumentError => e
{
status: 'failed',
recoverable: true,
errors: { base: [e.message] },
base_errors: [] }
rescue StandardError => e
Rails.logger.error(
"post_bulk_creator_failure #{ { error: e.class.name,
message: e.message }.to_json }")
{
status: 'failed',
recoverable: false,
errors: { base: ['登録中にエラーが発生しました.'] },
base_errors: [] }
end
def normalised_attributes attributes, preflight, index
{
url: preflight[:url],
title: preflight[:title],
thumbnail_base: preflight[:thumbnail_base],
thumbnail: thumbnail_for(index, attributes),
tags: preflight[:tags],
parent_post_ids: preflight[:parent_post_ids],
original_created_from: preflight[:original_created_from],
original_created_before: preflight[:original_created_before],
duration: preflight[:duration],
video_ms: preflight[:video_ms],
direct_tag_specs: preflight[:direct_tag_specs],
default_tag_specs: preflight[:default_tag_specs],
snapshot_tag_specs: preflight[:snapshot_tag_specs],
post_tag_specs: preflight[:post_tag_specs],
tag_sections: preflight[:tag_sections],
normalised_parent_post_ids: preflight[:normalised_parent_post_ids] }
end
def thumbnail_for index, attributes
return nil if attributes['thumbnail_base'].present? || attributes[:thumbnail_base].present?
@thumbnails[index]
end
def existing_post_for_race attributes, record = nil
return nil if record.present? && !(record.errors.of_kind?(:url, :taken))
normal_url = PostUrlNormaliser.normalise(attributes['url'] || attributes[:url])
return nil if normal_url.blank?
compact_existing_post(Post.with_attached_thumbnail.find_by(url: normal_url))
end
def compact_existing_post post
PostCompactRepr.base(post)
end
end
+257
ファイルの表示
@@ -0,0 +1,257 @@
class PostCreatePlan
VIDEO_TAG_NAME = '動画'.freeze
TAGME_TAG_NAME = 'タグ希望'.freeze
NO_DEERJIKIST_TAG_NAME = 'ニジラー情報不詳'.freeze
def initialize attributes:
@attributes = attributes.symbolize_keys
@existing_tags_by_name = nil
end
def build!
direct_tag_specs, tag_sections = parse_direct_tag_specs
default_tag_specs = build_default_tag_specs(direct_tag_specs)
snapshot_tag_specs = merge_tag_specs(direct_tag_specs + default_tag_specs)
preload_existing_tags_by_name!(snapshot_tag_specs.map { _1[:name] })
validate_new_tag_specs!(snapshot_tag_specs)
post_tag_specs = expand_parent_tag_specs(snapshot_tag_specs)
video_ms = normalise_video_ms(snapshot_tag_specs)
validate_video_sections!(video_ms, tag_sections)
parent_post_ids = normalise_parent_post_ids
validate_parent_post_ids!(parent_post_ids)
{
url: @attributes[:url],
title: @attributes[:title].to_s,
thumbnail_base: @attributes[:thumbnail_base].presence,
original_created_from: @attributes[:original_created_from].presence,
original_created_before: @attributes[:original_created_before].presence,
tags: serialised_tags(direct_tag_specs, tag_sections),
display_tags: display_tags(direct_tag_specs, tag_sections),
duration: @attributes[:duration].to_s,
video_ms: video_ms,
parent_post_ids: parent_post_ids.join(' '),
direct_tag_specs: direct_tag_specs,
default_tag_specs: default_tag_specs,
snapshot_tag_specs: snapshot_tag_specs,
post_tag_specs: post_tag_specs,
tag_sections: tag_sections,
normalised_parent_post_ids: parent_post_ids }
end
private
def tag_names = @attributes[:tags].to_s.split
def parse_direct_tag_specs
tag_sections = { }
direct_tag_specs = []
tag_names.each do |raw_name|
tag_name, category, sections = parse_raw_tag_name(raw_name)
existing_tag = existing_tags_by_name[tag_name]
raise Tag::NicoTagNormalisationError if existing_tag&.nico?
raise Tag::DeprecatedTagNormalisationError, [existing_tag.name] if existing_tag&.deprecated?
direct_tag_specs << {
name: tag_name,
category: (category || existing_tag&.category || 'general').to_sym }
if sections.present?
tag_sections[tag_name] ||= []
tag_sections[tag_name].concat(sections)
tag_sections[tag_name] = Tag.merge_section_ranges(tag_sections[tag_name])
tag_sections.delete(tag_name) if tag_sections[tag_name] == [[0, nil]]
end
end
[merge_tag_specs(direct_tag_specs), tag_sections]
end
def parse_raw_tag_name raw_name
name = raw_name.to_s
prefix, category =
Tag::CATEGORY_PREFIXES.find {
name.downcase.start_with?(_1[0])
} || ['', nil]
name = name.sub(/\A#{ prefix }/i, '')
sections = []
while (match = name.match(/\A(\S*?)\[([^\[\]\s]*)-([^\[\]\s]*)\](\S*)\z/))
name = "#{ match[1] }#{ match[4] }"
next if match[2].empty? && match[3].empty?
sections << Tag.normalise_section_range!(
begin_raw: match[2],
end_raw: match[3],
tag_name: name)
end
raise Tag::SectionLiteralParseError.new(raw_name, raw_name) if name.include?('[') || name.include?(']')
[TagName.canonicalise(name).first, category&.to_sym, sections]
end
def build_default_tag_specs direct_tag_specs
default_tag_specs = []
if direct_tag_specs.length < 10 && direct_tag_specs.none? { _1[:name] == TAGME_TAG_NAME }
default_tag_specs << {
name: TAGME_TAG_NAME,
category: :meta }
end
if direct_tag_specs.none? { deerjikist_tag_spec?(_1) }
default_tag_specs << {
name: NO_DEERJIKIST_TAG_NAME,
category: :meta }
end
default_tag_specs
end
def validate_new_tag_specs! specs
Array(specs).each do |spec|
next if existing_tags_by_name.key?(spec[:name])
validate_new_tag_spec!(spec)
end
end
def validate_new_tag_spec! spec
tag_name = TagName.new(name: spec[:name])
tag = Tag.new(category: spec[:category], tag_name:)
return if tag_name.valid? && tag.valid?
post = Post.new
tag_name.errors[:name].each do |message|
post.errors.add :tags, "#{ spec[:name] }: #{ message }"
end
tag.errors.each do |error|
next if error.attribute == :tag_name
post.errors.add :tags, "#{ spec[:name] }: #{ error.message }"
end
raise ActiveRecord::RecordInvalid, post
end
def expand_parent_tag_specs snapshot_tag_specs
existing_snapshot_tags = snapshot_tag_specs.filter_map { existing_tags_by_name[_1[:name]] }
expanded_parent_specs =
Tag.expand_parent_tags(existing_snapshot_tags)
.reject(&:deprecated?)
.map { |tag|
{
name: tag.name,
category: tag.category.to_sym } }
merge_tag_specs(snapshot_tag_specs + expanded_parent_specs)
end
def merge_tag_specs specs
specs.each_with_object({ }) do |spec, merged|
merged[spec[:name]] =
if merged.key?(spec[:name]) && merged[spec[:name]][:category] != :general
merged[spec[:name]]
else
{
name: spec[:name],
category: spec[:category] }
end
end.values.sort_by { _1[:name] }
end
def existing_tags_by_name
@existing_tags_by_name ||= begin
names = tag_names.map { canonical_tag_name_without_sections(_1) }.uniq
Tag.joins(:tag_name).where(tag_names: { name: names }).index_by(&:name)
end
end
def preload_existing_tags_by_name! names
wanted_names = Array(names).map { _1.to_s }.reject(&:blank?).uniq
missing_names = wanted_names - existing_tags_by_name.keys
return if missing_names.empty?
existing_tags_by_name.merge!(
Tag.joins(:tag_name)
.where(tag_names: { name: missing_names })
.index_by(&:name))
end
def canonical_tag_name_without_sections raw_name
name, = parse_raw_tag_name(raw_name)
name
end
def deerjikist_tag_spec? spec
return true if spec[:category] == :deerjikist
existing_tags_by_name[spec[:name]]&.deerjikist?
end
def normalise_parent_post_ids
Array(@attributes[:parent_post_ids]).flat_map { _1.to_s.split }.map { |token|
id = Integer(token, exception: false)
raise ArgumentError, "親投稿 Id. が不正です: #{ token }" if id.nil? || id <= 0
id
}.uniq.sort
end
def validate_parent_post_ids! ids
missing = ids - Post.where(id: ids).pluck(:id)
raise ArgumentError, "存在しない親投稿 Id. があります: #{ missing.join(' ') }" if missing.present?
end
def serialised_tags direct_tag_specs, tag_sections
direct_tag_specs.map { |spec|
"#{ spec[:name] }#{ tag_sections[spec[:name]].to_a.map { Post.section_literal(_1) }.join }"
}.sort.join(' ')
end
def display_tags direct_tag_specs, tag_sections
direct_tag_specs.map { |spec|
{
name: spec[:name],
category: spec[:category].to_s,
section_literals: tag_sections[spec[:name]].to_a.map { Post.section_literal(_1) } }
}.sort_by { _1[:name] }
end
def normalise_video_ms snapshot_tag_specs
return nil unless snapshot_tag_specs.any? { _1[:name] == VIDEO_TAG_NAME }
video_ms = @attributes[:video_ms]
if video_ms.present?
value = Integer(video_ms, exception: false)
raise PostCreator::VideoMsParseError unless value&.positive?
return value
end
duration = @attributes[:duration]
return nil if duration.blank?
value = Tag.time_to_ms!(duration.to_s, tag_name: '動画時間')
raise PostCreator::VideoMsParseError unless value.positive?
value
rescue Tag::SectionLiteralParseError
raise PostCreator::VideoMsParseError
end
def validate_video_sections! video_ms, tag_sections
return unless video_ms
tag_sections.each_value do |ranges|
ranges.each do |begin_ms, end_ms|
if begin_ms >= video_ms
post = Post.new
post.errors.add :video_ms, 'タグ区間の開始が動画時間以上です.'
raise ActiveRecord::RecordInvalid, post
end
if end_ms && end_ms > video_ms
post = Post.new
post.errors.add :video_ms, 'タグ区間の終端が動画時間を超えてゐます.'
raise ActiveRecord::RecordInvalid, post
end
end
end
end
end
+140
ファイルの表示
@@ -0,0 +1,140 @@
class PostCreatePreflight
class ValidationFailed < StandardError
attr_reader :fields, :base_errors
def initialize fields: { }, base_errors: []
super('入力内容を確認してください.')
@fields = fields
@base_errors = base_errors
end
end
def initialize attributes:, thumbnail: nil
@attributes = attributes.symbolize_keys
@thumbnail = thumbnail
end
def run
preview = PostImportPreviewer.new.preview_rows(
rows: [preview_row],
fetch_metadata: false).first
if preview[:existing_post_id].present?
return {
url: preview[:url],
title: preview[:attributes]['title'],
thumbnail_base: preview[:attributes]['thumbnail_base'],
tags: preview[:attributes]['tags'],
parent_post_ids: preview[:attributes]['parent_post_ids'],
original_created_from: preview[:attributes]['original_created_from'],
original_created_before: preview[:attributes]['original_created_before'],
duration: preview[:attributes]['duration'],
video_ms: preview[:attributes]['video_ms'],
display_tags: preview[:display_tags] || [],
field_warnings: final_field_warnings(preview[:field_warnings] || { }),
base_warnings: preview[:base_warnings],
existing_post_id: preview[:existing_post_id],
existing_post: existing_post_compact(preview[:existing_post_id]) }
end
if preview[:validation_errors].present?
raise ValidationFailed.new(fields: preview[:validation_errors])
end
validate_thumbnail_upload!
plan = PostCreatePlan.new(
attributes: {
url: preview[:url],
title: preview[:attributes]['title'],
thumbnail_base: preview[:attributes]['thumbnail_base'],
tags: preview[:attributes]['tags'],
parent_post_ids: preview[:attributes]['parent_post_ids'],
original_created_from: preview[:attributes]['original_created_from'],
original_created_before: preview[:attributes]['original_created_before'],
duration: preview[:attributes]['duration'],
video_ms: preview[:attributes]['video_ms'] }).build!
{
url: plan[:url],
title: plan[:title],
thumbnail_base: plan[:thumbnail_base],
tags: plan[:tags],
parent_post_ids: plan[:parent_post_ids],
original_created_from: plan[:original_created_from],
original_created_before: plan[:original_created_before],
duration: plan[:duration],
video_ms: plan[:video_ms],
display_tags: plan[:display_tags],
direct_tag_specs: plan[:direct_tag_specs],
default_tag_specs: plan[:default_tag_specs],
snapshot_tag_specs: plan[:snapshot_tag_specs],
post_tag_specs: plan[:post_tag_specs],
tag_sections: plan[:tag_sections],
normalised_parent_post_ids: plan[:normalised_parent_post_ids],
field_warnings: final_field_warnings(preview[:field_warnings] || { }),
base_warnings: preview[:base_warnings],
existing_post_id: preview[:existing_post_id],
existing_post: existing_post_compact(preview[:existing_post_id]) }
end
private
def preview_row
{
source_row: 1,
url: @attributes[:url].to_s,
attributes: {
'title' => @attributes[:title].to_s,
'thumbnail_base' => @attributes[:thumbnail_base].to_s,
'original_created_from' => @attributes[:original_created_from].to_s,
'original_created_before' => @attributes[:original_created_before].to_s,
'duration' => @attributes[:duration].to_s,
'video_ms' => @attributes[:video_ms],
'tags' => @attributes[:tags].to_s,
'parent_post_ids' => parent_post_ids_text },
provenance: {
'url' => 'manual',
'title' => 'manual',
'thumbnail_base' => 'manual',
'original_created_from' => 'manual',
'original_created_before' => 'manual',
'duration' => 'manual',
'video_ms' => 'manual',
'tags' => 'manual',
'parent_post_ids' => 'manual' },
tag_sources: {
'automatic' => '',
'manual' => @attributes[:tags].to_s } }
end
def parent_post_ids_text
Array(@attributes[:parent_post_ids]).flat_map { _1.to_s.split }.join(' ')
end
def validate_thumbnail_upload!
return if @attributes[:thumbnail_base].present?
return if @thumbnail.blank?
PostThumbnailUploadValidator.validate!(@thumbnail)
rescue PostThumbnailUploadValidator::InvalidUpload => e
raise ValidationFailed.new(fields: { thumbnail: [e.message] })
end
def existing_post_compact post_id
return nil if post_id.blank?
post = Post.with_attached_thumbnail.find_by(id: post_id)
PostCompactRepr.base(post)
end
def final_field_warnings field_warnings
thumbnail_warnings = (field_warnings['thumbnail_base'] || []).reject { _1 == 'サムネールなし' }
if @attributes[:thumbnail_base].blank? && @thumbnail.blank?
thumbnail_warnings = (thumbnail_warnings + ['サムネールなし']).uniq
end
next_warnings = field_warnings.except('thumbnail_base')
next_warnings['thumbnail_base'] = thumbnail_warnings if thumbnail_warnings.present?
next_warnings
end
end
+154
ファイルの表示
@@ -0,0 +1,154 @@
class PostCreator
class VideoMsParseError < ArgumentError; end
attr_reader :field_warnings
def initialize actor:, attributes:
@actor = actor
@attributes = attributes.symbolize_keys
@field_warnings = { }
end
def create!
thumbnail_attachment = prepare_thumbnail_attachment
post = Post.new(title: @attributes[:title].presence,
url: @attributes[:url],
thumbnail_base: @attributes[:thumbnail_base].presence,
uploaded_user: @actor,
original_created_from: @attributes[:original_created_from].presence,
original_created_before: @attributes[:original_created_before].presence)
ApplicationRecord.transaction do
post.save!
post.thumbnail.attach(thumbnail_attachment) if thumbnail_attachment.present?
snapshot_tags = planned_snapshot_tags
post_tags = planned_post_tags
sections = planned_sections
TagVersioning.record_tag_snapshots!(snapshot_tags, created_by_user: @actor)
post.video_ms = planned_video_ms
post.save!
sync_post_tags!(post, post_tags, sections)
sync_parent_posts!(post, planned_parent_post_ids)
PostVersionRecorder.record!(post:, event_type: :create, created_by_user: @actor)
end
post
rescue StandardError
post&.thumbnail&.purge if post&.thumbnail&.attached?
raise
end
private
def prepare_thumbnail_attachment
PostThumbnailAttachmentBuilder.build(
thumbnail: @attributes[:thumbnail],
thumbnail_base: @attributes[:thumbnail_base].presence)
end
def planned_snapshot_tags = planned_create_attributes[:snapshot_tags]
def planned_post_tags = planned_create_attributes[:post_tags]
def planned_sections = planned_create_attributes[:tag_sections]
def planned_parent_post_ids = planned_create_attributes[:normalised_parent_post_ids]
def planned_video_ms
planned_create_attributes[:video_ms]
end
def planned_create_attributes
@planned_create_attributes ||= begin
if @attributes.key?(:snapshot_tag_specs)
snapshot_tags = materialise_tags(@attributes[:snapshot_tag_specs] || [])
post_tags = materialise_tags(@attributes[:post_tag_specs] || [])
{
snapshot_tags: snapshot_tags,
post_tags: post_tags,
tag_sections: materialise_sections(
@attributes[:tag_sections] || { },
snapshot_tags,
post_tags),
normalised_parent_post_ids: @attributes[:normalised_parent_post_ids] || [],
video_ms: @attributes[:video_ms] }
else
build_materialised_plan
end
end
end
def build_materialised_plan
plan = PostCreatePlan.new(attributes: @attributes).build!
snapshot_tags = materialise_tags(plan[:snapshot_tag_specs] || [])
post_tags = materialise_tags(plan[:post_tag_specs] || [])
{
snapshot_tags: snapshot_tags,
post_tags: post_tags,
tag_sections: materialise_sections(
plan[:tag_sections] || { },
snapshot_tags,
post_tags),
normalised_parent_post_ids: plan[:normalised_parent_post_ids] || [],
video_ms: plan[:video_ms] }
end
def materialise_tags specs
Array(specs).each_with_object({ }) do |spec, tags|
name = spec[:name] || spec['name']
category = spec[:category] || spec['category']
next if name.blank? || category.blank?
tag = Tag.find_or_create_by_tag_name!(name, category:)
tag.update!(category:) if tag.category.to_sym != category.to_sym
tags[name] ||= tag
end.values
end
def materialise_sections sections_by_name, snapshot_tags, post_tags
tags_by_name = post_tags.index_by(&:name)
snapshot_tags.each do |tag|
tags_by_name[tag.name] ||= tag
end
sections_by_name.each_with_object({ }) do |(tag_name, ranges), sections|
tag = tags_by_name[tag_name.to_s]
next if tag.nil?
sections[tag.id] = Array(ranges).map { |range| [range[0], range[1]] }
end
end
def sync_post_tags! post, desired_tags, sections
desired_ids = desired_tags.map(&:id).to_set
current_ids = post.tags.pluck(:id).to_set
Tag.where(id: desired_ids - current_ids).find_each do |tag|
PostTag.create_or_find_by!(post:, tag:, created_user: @actor)
end
PostTagSection.where(post_id: post.id).destroy_all
sections.each do |tag_id, ranges|
ranges.each do |begin_ms, end_ms|
PostTagSection.create!(post_id: post.id,
tag_id:,
begin_ms:,
end_ms:)
end
end
PostTag.where(post_id: post.id,
tag_id: (current_ids - desired_ids).to_a).kept.find_each do |post_tag|
post_tag.discard_by!(@actor)
end
end
def sync_parent_posts! post, ids
if ids.include?(post.id)
post.errors.add :parent_post_ids, '自分自身を親投稿にはできません.'
raise ActiveRecord::RecordInvalid, post
end
missing = ids - Post.where(id: ids).pluck(:id)
if missing.present?
post.errors.add :parent_post_ids, "存在しない親投稿 Id. があります: #{ missing.join(' ') }"
raise ActiveRecord::RecordInvalid, post
end
ids.each { |parent_post_id| PostImplication.create_or_find_by!(post:, parent_post_id:) }
end
end
+528
ファイルの表示
@@ -0,0 +1,528 @@
require 'time'
require 'timeout'
class PostImportPreviewer
FIELDS = [
'title',
'thumbnail_base',
'original_created_from',
'original_created_before',
'video_ms',
'duration',
'tags',
'parent_post_ids'].freeze
FETCH_WARNING_FIELDS = ['url', 'title', 'thumbnail_base'].freeze
TITLE_FETCH_WARNING = 'タイトルを取得できませんでした.'.freeze
THUMBNAIL_FETCH_WARNING = 'サムネールを取得できませんでした.'.freeze
METADATA_FETCH_WARNING = '自動取得に失敗しました.'.freeze
def preview_rows rows:, fetch_metadata: true, metadata_cache: { }
prepared_rows = rows.map { prepare_row(_1) }
url_counts = prepared_rows.filter_map { _1[:normal_url] }.tally
existing_posts =
Post.where(url: prepared_rows.map { _1[:normal_url] }.compact.uniq).index_by(&:url)
existing_parent_ids = preload_parent_ids(prepared_rows)
preload_metadata!(
prepared_rows,
fetch_metadata,
metadata_cache,
existing_posts,
url_counts)
known_tags =
preload_known_tags(
prepared_rows,
fetch_metadata,
metadata_cache,
existing_posts,
url_counts)
prepared_rows.map { |row|
preview_row(row,
fetch_metadata:,
metadata_cache:,
existing_posts:,
url_counts:,
known_tags:,
existing_parent_ids:)
}
end
def normalised_url value
PostUrlNormaliser.normalise(value)
end
private
def prepare_row row
source = row.symbolize_keys
url = source[:url].to_s.strip
normal_url = normalised_url(url)
source.merge(url_text: url, normal_url:, url_error: validate_url_safety(normal_url))
end
def preview_row row,
fetch_metadata:,
metadata_cache:,
existing_posts:,
url_counts:,
known_tags:,
existing_parent_ids:
attributes = initial_attributes(row)
provenance = initial_provenance(row)
tag_sources = initial_tag_sources(row, attributes, provenance)
field_warnings = initial_field_warnings(row)
base_warnings = initial_base_warnings(row)
url = row[:url_text]
provenance['url'] = 'manual'
normal_url = row[:normal_url]
url_for_metadata = normal_url || url
existing_post = normal_url.present? ? existing_posts[normal_url] : nil
validation_errors = {}
validation_errors[:url] = ['URL が不正です.'] if normal_url.blank?
if row[:url_error].present?
validation_errors[:url] = [row[:url_error]]
end
if normal_url.present? && url_counts[normal_url].to_i > 1
validation_errors[:url] = ['URL が重複しています.']
end
if row[:metadata_url].present? && row[:metadata_url] != url_for_metadata
clear_automatic_values!(attributes, provenance, tag_sources)
field_warnings = { }
base_warnings = [ ]
end
if validation_errors.blank? && normal_url.present? && existing_post
attributes['tags'] = merged_tags(tag_sources, provenance['tags'])
warnings_present = field_warnings.values.any?(&:present?) || base_warnings.present?
return { source_row: row[:source_row],
url: normal_url,
attributes:,
provenance:,
tag_sources:,
metadata_url: url_for_metadata,
skip_reason: 'existing',
existing_post_id: existing_post.id,
field_warnings:,
base_warnings:,
validation_errors:,
status: warnings_present ? 'warning' : 'ready' }
end
should_fetch = validation_errors.blank?
should_fetch &&= should_fetch_metadata?(fetch_metadata, row[:source_row].to_i)
if should_fetch
clear_fetch_warnings!(field_warnings)
metadata = metadata_for(url_for_metadata, metadata_cache)
if metadata[:validation_errors].present?
validation_errors.merge!(metadata[:validation_errors])
else
apply_metadata!(attributes, provenance, tag_sources, metadata[:data])
apply_fetch_warnings!(field_warnings, metadata[:warnings])
end
end
attributes['url'] = url
validate_basic_data(attributes, validation_errors)
validate_preview_tags(merged_tags(tag_sources, provenance['tags']),
validation_errors,
field_warnings,
known_tags)
validate_parents(attributes['parent_post_ids'], validation_errors, existing_parent_ids)
attributes.delete('url')
attributes['tags'] = merged_tags(tag_sources, provenance['tags'])
warnings_present = field_warnings.values.any?(&:present?) || base_warnings.present?
{ source_row: row[:source_row],
url: validation_errors[:url].present? ? url : (normal_url || url),
attributes:,
provenance:,
tag_sources:,
metadata_url: url_for_metadata,
skip_reason: nil,
existing_post_id: nil,
field_warnings:,
base_warnings:,
validation_errors:,
status: validation_errors.present? ? 'error' : (warnings_present ? 'warning' : 'ready') }
end
def should_fetch_metadata? fetch_metadata, source_row
case fetch_metadata
when true then true
when Integer then fetch_metadata == source_row
when false, nil then false
else raise ArgumentError, '取得対象が不正です.'
end
end
def initial_attributes row
attributes = row[:attributes]&.stringify_keys || { }
FIELDS.to_h { |field|
value = attributes[field]
normalised =
if field == 'duration'
normalise_duration_attribute(value)
elsif field == 'video_ms'
value.nil? ? '' : value.to_s
else
value.to_s
end
[field, normalised] }
end
def initial_provenance row
provenance = row[:provenance]&.stringify_keys || { }
FIELDS.to_h { |field| [field, provenance[field].presence || 'automatic'] }
.merge('url' => 'manual')
end
def initial_tag_sources row, attributes, provenance
sources = row[:tag_sources]&.stringify_keys || { 'automatic' => '', 'manual' => '' }
sources['automatic'] = sources['automatic'].to_s
sources['manual'] =
provenance['tags'] == 'manual' ? attributes['tags'].to_s : sources['manual'].to_s
sources
end
def initial_field_warnings row
(row[:field_warnings] || { })
.stringify_keys
.transform_values { |value| Array(value).map(&:to_s) }
end
def initial_base_warnings row
Array(row[:base_warnings]).map(&:to_s)
end
def clear_automatic_values! attributes, provenance, tag_sources
['title', 'thumbnail_base', 'original_created_from',
'original_created_before', 'video_ms', 'duration'].each do |field|
attributes[field] = '' if provenance[field] == 'automatic'
end
tag_sources['automatic'] = ''
attributes['tags'] = merged_tags(tag_sources, provenance['tags'])
end
def clear_fetch_warnings! field_warnings
FETCH_WARNING_FIELDS.each do |field|
field_warnings.delete(field)
end
end
def metadata_for url, cache
cache[url] ||= fetch_metadata(url)
end
def fetch_metadata url
return { data: { }, warnings: { 'url' => ['URL が空です.'] },
validation_errors: { } } if url.blank?
data = sanitise_metadata(PostMetadataFetcher.fetch(url).stringify_keys.compact)
warnings = { }
add_field_warning!(warnings, 'title', TITLE_FETCH_WARNING) if data['title'].blank?
if data['thumbnail_base'].blank?
add_field_warning!(warnings, 'thumbnail_base', THUMBNAIL_FETCH_WARNING)
end
{ data:, warnings:, validation_errors: { } }
rescue Preview::UrlSafety::UnsafeUrl => e
Rails.logger.info(
"post_import_metadata_fetch_unsafe_url "\
"#{ { error: e.class.name, message: e.message }.to_json }")
{ data: { }, warnings: { }, validation_errors: { url: [e.message] } }
rescue Preview::HttpFetcher::FetchFailed,
Preview::HttpFetcher::ResponseTooLarge => e
Rails.logger.info(
"post_import_metadata_fetch_failure "\
"#{ { error: e.class.name, message: e.message }.to_json }")
{ data: { }, warnings: { 'url' => [METADATA_FETCH_WARNING] }, validation_errors: { } }
end
def sanitise_metadata metadata
{
'title' => sanitise_metadata_title(metadata['title']),
'thumbnail_base' => sanitise_metadata_url(metadata['thumbnail_base']),
'original_created_from' => sanitise_metadata_time(metadata['original_created_from']),
'original_created_before' => sanitise_metadata_time(metadata['original_created_before']),
'duration' => sanitise_metadata_duration(metadata['duration']),
'tags' => metadata['tags'].to_s.presence }.compact
end
def sanitise_metadata_title value
value.is_a?(String) ? value.presence : nil
end
def sanitise_metadata_url value
return nil unless value.is_a?(String)
stripped = value.strip
return nil if stripped.blank?
uri = URI.parse(stripped)
return nil unless uri.is_a?(URI::HTTP) && uri.host.present?
stripped
rescue URI::InvalidURIError
nil
end
def sanitise_metadata_time value
return nil unless value.is_a?(String)
Time.iso8601(value).in_time_zone.change(sec: 0, nsec: 0).iso8601
rescue ArgumentError, TypeError
nil
end
def sanitise_metadata_duration value
return nil unless value.is_a?(String)
value.presence
end
def normalise_duration_attribute value
return '' if value.nil?
return value if value.is_a?(String)
milliseconds = Integer(value, exception: false)
return value.to_s if milliseconds.nil? || milliseconds <= 0
seconds_string = (milliseconds / 1_000.0).to_s
seconds_string.end_with?('.0') ? seconds_string.delete_suffix('.0') : seconds_string
end
def preload_metadata! prepared_rows, fetch_metadata, metadata_cache, existing_posts, url_counts
urls = prepared_rows.filter_map { |row|
next unless row[:normal_url].present?
next if row[:url_error].present?
next unless should_fetch_metadata?(fetch_metadata, row[:source_row].to_i)
next if url_counts[row[:normal_url]].to_i > 1
next if existing_posts.key?(row[:normal_url])
row[:normal_url]
}.uniq
urls = urls.reject { metadata_cache.key?(_1) }
return if urls.empty?
url_queue = Queue.new
result_queue = Queue.new
urls.each { url_queue << _1 }
workers = [urls.length, 4].min.times.map {
Thread.new {
Rails.application.executor.wrap do
loop do
url = url_queue.pop(true)
result_queue << [url, safe_fetch_metadata(url)]
rescue ThreadError
break
end
end
}
}
Timeout.timeout(15) { workers.each(&:join) }
rescue Timeout::Error
workers&.each(&:kill)
ensure
workers&.each(&:join)
while result_queue&.size.to_i.positive?
url, result = result_queue.pop
metadata_cache[url] = result
end
urls&.each do |url|
metadata_cache[url] ||= { data: { }, warnings: { 'url' => [METADATA_FETCH_WARNING] } }
end
end
def safe_fetch_metadata url
fetch_metadata(url)
rescue Preview::UrlSafety::UnsafeUrl => e
Rails.logger.info(
"post_import_metadata_fetch_unsafe_url "\
"#{ { error: e.class.name, message: e.message }.to_json }")
{ data: { }, warnings: { }, validation_errors: { url: [e.message] } }
rescue StandardError => e
Rails.logger.error(
"post_import_metadata_fetch_unexpected_failure "\
"#{ { error: e.class.name, message: e.message }.to_json }")
{ data: { }, warnings: { 'url' => [METADATA_FETCH_WARNING] }, validation_errors: { } }
end
def preload_known_tags prepared_rows, fetch_metadata, metadata_cache, existing_posts, url_counts
names = prepared_rows.flat_map { |row|
attributes = initial_attributes(row)
provenance = initial_provenance(row)
tag_sources = initial_tag_sources(row, attributes, provenance)
if metadata_url_changed?(row)
clear_automatic_values!(attributes, provenance, tag_sources)
end
if should_apply_metadata_to_row?(row, fetch_metadata, existing_posts, url_counts)
metadata = metadata_for(row[:normal_url], metadata_cache)
apply_metadata!(attributes, provenance, tag_sources, metadata[:data])
end
preview_tag_names(merged_tags(tag_sources, provenance['tags']))
}.compact.uniq
return { } if names.empty?
Tag.joins(:tag_name)
.where(tag_names: { name: names })
.includes(:tag_name)
.to_a
.index_by(&:name)
end
def preload_parent_ids prepared_rows
ids = prepared_rows.flat_map { |row|
attributes = initial_attributes(row)
preview_parent_ids(attributes['parent_post_ids'])
}.uniq
return { } if ids.empty?
Post.where(id: ids).pluck(:id).to_h { [_1, true] }
end
def metadata_url_changed? row
row[:metadata_url].present? && row[:metadata_url] != (row[:normal_url] || row[:url_text])
end
def should_apply_metadata_to_row? row, fetch_metadata, existing_posts, url_counts
normal_url = row[:normal_url]
return false if normal_url.blank?
return false if row[:url_error].present?
return false if url_counts[normal_url].to_i > 1
return false if existing_posts.key?(normal_url)
should_fetch_metadata?(fetch_metadata, row[:source_row].to_i)
end
def validate_url_safety normal_url
return nil if normal_url.blank?
Preview::UrlSafety.validate(normal_url)
nil
rescue Preview::UrlSafety::UnsafeUrl => e
e.message
end
def apply_metadata! attributes, provenance, tag_sources, metadata
metadata.each do |field, value|
if field == 'tags'
next if provenance['tags'] == 'manual'
tag_sources['automatic'] = value.to_s
attributes['tags'] = merged_tags(tag_sources)
next
end
next unless provenance[field] == 'automatic'
attributes[field] = value
provenance[field] = 'automatic'
end
end
def apply_fetch_warnings! field_warnings, warnings
warnings.each do |field, values|
values.each do |value|
add_field_warning!(field_warnings, field, value)
end
end
end
def add_field_warning! field_warnings, field, message
field_warnings[field] ||= []
field_warnings[field] << message unless field_warnings[field].include?(message)
end
def merged_tags sources, origin = nil
return sources['manual'].to_s if origin == 'manual'
sources['automatic'].to_s
end
def preview_tag_names raw
names = raw.to_s.split
return [] if names.empty?
if names.any? { _1.downcase.start_with?('nico:') }
return []
end
names.map { |name| TagName.canonicalise(name.sub(/\[.*\]\z/, '')).first }
rescue Tag::SectionLiteralParseError
[]
end
def validate_preview_tags raw, errors, field_warnings, known_tags
names = raw.to_s.split
return if names.empty?
if names.any? { _1.downcase.start_with?('nico:') }
errors[:tags] = ['ニコニコ・タグは直接指定できません.']
return
end
parsed = names.map { |name| TagName.canonicalise(name.sub(/\[.*\]\z/, '')).first }
existing = parsed.filter_map { known_tags[_1] }
deprecated = existing.select(&:deprecated?).map(&:name)
errors[:tags] = ["廃止済みタグがあります: #{ deprecated.join(' ') }"] if deprecated.present?
known = existing.reject(&:deprecated?).map(&:name)
new_tags = parsed.uniq - known
if new_tags.present?
add_field_warning!(field_warnings, 'tags', "新規タグを作成します: #{ new_tags.join(' ') }")
end
rescue Tag::SectionLiteralParseError
errors[:tags] = ['タグ区間の記法が不正です.']
end
def validate_basic_data attributes, errors
post = Post.new(title: attributes['title'].presence,
url: attributes['url'],
thumbnail_base: attributes['thumbnail_base'].presence,
original_created_from: attributes['original_created_from'].presence,
original_created_before: attributes['original_created_before'].presence,
video_ms: parse_video_ms(attributes, errors))
post.valid?
post.errors.each do |error|
next if error.attribute == :url && error.type == :taken
(errors[error.attribute] ||= []) << error.message
end
end
def parse_video_ms attributes, errors
return nil unless attributes['tags'].to_s.split.include?('動画')
video_ms = attributes['video_ms']
if video_ms.present?
value = Integer(video_ms, exception: false)
if value&.positive?
return value
end
errors[:video_ms] = ['動画時間の記法が不正です.']
return nil
end
value = attributes['duration']
return nil if value.blank?
Tag.time_to_ms!(value.to_s, tag_name: '動画時間')
rescue Tag::SectionLiteralParseError
errors[:video_ms] = ['動画時間の記法が不正です.']
nil
end
def preview_parent_ids raw
raw.to_s.split.map { Integer(_1, exception: false) }.compact
end
def validate_parents raw, errors, existing_parent_ids
ids = raw.to_s.split.map { Integer(_1, exception: false) }
return if ids.compact.empty? && raw.to_s.blank?
if ids.any? { _1.nil? || _1 <= 0 }
errors[:parent_post_ids] = ['親投稿 Id. が不正です.']
return
end
if ids.uniq.any? { !existing_parent_ids[_1] }
errors[:parent_post_ids] = ['存在しない親投稿 Id. があります.']
end
end
end
+199
ファイルの表示
@@ -0,0 +1,199 @@
require 'time'
require 'date'
class PostMetadataFetcher
TIMESTAMP_PATTERN =
Regexp.new(
'\A(\d{4})-(\d{2})-(\d{2})T(\d{2})' \
'(?::(\d{2})(?::(\d{2})(?:\.(\d+))?)?)?' \
'(Z|[+-]\d{2}:?\d{2})?\z')
def self.fetch raw_url
uri, = Preview::UrlSafety.validate(raw_url)
response = Preview::HttpFetcher.fetch(
uri.to_s,
max_bytes: Preview::ThumbnailFetcher::HTML_MAX_BYTES)
metadata = Preview::HtmlMetadataExtractor.extract(response)
document = Nokogiri::HTML.parse(response.body)
content = lambda { |name|
document
.at_css("meta[property='#{ name }'], meta[name='#{ name }']")
&.[]('content')
&.strip
&.presence
}
duration = content.call('og:video:duration') || content.call('video:duration')
published = content.call('article:published_time') || content.call('date')
created_range = original_created_range(published)
platform_tags = platform_tags(uri)
{ title: metadata[:title],
thumbnail_base:
Preview::KnownSiteExtractor.thumbnail_url(uri) || metadata[:image_url],
original_created_from: serialise_time(created_range&.first),
original_created_before: serialise_time(created_range&.last),
duration: serialise_duration(duration),
display_tags: display_tags(platform_tags),
tags: platform_tags.join(' ') }
end
def self.platform_tags uri
return ['動画', 'YouTube'] if Preview::KnownSiteExtractor.youtube_video_id(uri)
return ['動画', 'ニコニコ'] if Preview::KnownSiteExtractor.niconico_video_id(uri)
[]
end
def self.display_tags names
return [] if names.blank?
existing_tags =
Tag
.joins(:tag_name)
.where(tag_names: { name: names })
.index_by(&:name)
names.map { |name|
tag = existing_tags[name]
{
name: name,
category: (tag&.category || 'meta'),
section_literals: [] }
}
end
def self.original_created_range value
return nil if value.blank?
raw = value.to_s.strip
from, before =
case raw
when /\A(\d{4})\z/
year = Regexp.last_match(1).to_i
from = Time.zone.local(year, 1, 1)
[from, from + 1.year]
when /\A(\d{4})-(\d{2})\z/
year = Regexp.last_match(1).to_i
month = Regexp.last_match(2).to_i
return nil unless month.between?(1, 12)
from = Time.zone.local(year, month, 1)
[from, from + 1.month]
when /\A(\d{4})-(\d{2})-(\d{2})\z/
year = Regexp.last_match(1).to_i
month = Regexp.last_match(2).to_i
day = Regexp.last_match(3).to_i
return nil unless Date.valid_date?(year, month, day)
from = Time.zone.local(year, month, day)
[from, from + 1.day]
else
parse_timestamp_range(raw)
end
return nil if from.nil? || before.nil?
[from, before]
rescue ArgumentError, TypeError
nil
end
def self.parse_timestamp_range raw
match = raw.match(TIMESTAMP_PATTERN)
return nil unless match
year = match[1].to_i
month = match[2].to_i
day = match[3].to_i
hour = match[4].to_i
minute = match[5]&.to_i || 0
second = match[6]&.to_i || 0
fraction = match[7]
offset = match[8]
return nil unless valid_timestamp_components?(year, month, day, hour, minute, second)
return nil unless valid_offset?(offset)
nanoseconds = parse_nanoseconds(fraction)
timestamp =
if offset.present?
Time.new(
year,
month,
day,
hour,
minute,
second + Rational(nanoseconds, 1_000_000_000),
parse_offset(offset)).in_time_zone
else
Time.zone.local(year, month, day, hour, minute, second).change(nsec: nanoseconds)
end
from = timestamp.change(sec: 0, nsec: 0)
before =
if match[5].nil?
from + 1.hour
else
from + 1.minute
end
[from, before]
end
def self.parse_nanoseconds value
return 0 if value.blank?
digits = value[0, 9].ljust(9, '0')
Integer(digits, 10)
end
def self.parse_offset value
return '+00:00' if value == 'Z'
value.match?(/\A[+-]\d{2}:\d{2}\z/) ? value : "#{ value[0, 3] }:#{ value[3, 2] }"
end
def self.serialise_duration value
seconds = Float(value)
return nil unless seconds.positive?
milliseconds = (seconds * 1_000).round
seconds_string = (milliseconds / 1_000.0).to_s
seconds_string.end_with?('.0') ? seconds_string.delete_suffix('.0') : seconds_string
rescue ArgumentError, TypeError
nil
end
def self.valid_timestamp_components? year, month, day, hour, minute, second
return false unless Date.valid_date?(year, month, day)
return false unless hour.between?(0, 23)
return false unless minute.between?(0, 59)
return false unless second.between?(0, 59)
true
end
def self.valid_offset? value
return true if value.nil? || value == 'Z'
match = value.match(/\A([+-])(\d{2}):?(\d{2})\z/)
return false if match.nil?
hours = match[2].to_i
minutes = match[3].to_i
hours.between?(0, 23) && minutes.between?(0, 59)
end
def self.serialise_time value
return nil if value.nil?
value.nsec.zero? ? value.iso8601 : value.iso8601(9)
end
private_class_method :platform_tags,
:display_tags,
:original_created_range,
:parse_timestamp_range,
:parse_nanoseconds,
:parse_offset,
:serialise_duration,
:valid_timestamp_components?,
:valid_offset?,
:serialise_time
end
+11
ファイルの表示
@@ -0,0 +1,11 @@
class PostThumbnailAttachmentBuilder
def self.build thumbnail:, thumbnail_base:
if thumbnail_base.present?
return Post.remote_thumbnail_attachment(thumbnail_base)
end
return nil if thumbnail.blank?
Post.resized_thumbnail_attachment(thumbnail)
end
end
+34
ファイルの表示
@@ -0,0 +1,34 @@
class PostThumbnailUploadValidator
MAX_THUMBNAIL_BYTES = 20 * 1024 * 1024
ALLOWED_CONTENT_TYPES = Preview::ThumbnailFetcher::RASTER_IMAGE_CONTENT_TYPES.freeze
class InvalidUpload < StandardError; end
def self.validate! thumbnail
return if thumbnail.blank?
unless thumbnail.is_a?(ActionDispatch::Http::UploadedFile)
raise InvalidUpload, 'thumbnail upload が不正です.'
end
raise InvalidUpload, 'thumbnail file size が大きすぎます.' if thumbnail.size > MAX_THUMBNAIL_BYTES
raise InvalidUpload, 'サムネイル画像の形式が不正です.' unless allowed_content_type?(thumbnail.content_type)
bytes = thumbnail.read
raise InvalidUpload, 'サムネイル画像の形式が不正です.' if Post.svg_document_bytes?(bytes)
unless Post.raster_thumbnail_bytes?(bytes)
raise InvalidUpload, 'サムネイル画像の形式が不正です.'
end
attachment = Post.resized_thumbnail_attachment(
StringIO.new(bytes),
content_type: thumbnail.content_type)
attachment[:io].close if attachment[:io].respond_to?(:close)
rescue MiniMagick::Error, Timeout::Error
raise InvalidUpload, 'サムネイル画像の変換に失敗しました.'
ensure
thumbnail&.rewind if thumbnail.respond_to?(:rewind)
end
def self.allowed_content_type? content_type
mime_type = content_type.to_s.split(';', 2).first.to_s.downcase.strip
ALLOWED_CONTENT_TYPES.include?(mime_type)
end
end
+13
ファイルの表示
@@ -0,0 +1,13 @@
class PostUrlNormaliser
def self.normalise raw_url
value = raw_url.to_s.strip
uri = URI.parse(value)
return nil unless uri.is_a?(URI::HTTP) && uri.host.present?
uri.host = uri.host.downcase
uri.path = uri.path.sub(/\/\z/, '') if uri.path.present?
PostUrlSanitisationRule.sanitise(uri.to_s)
rescue URI::InvalidURIError
nil
end
end
+32
ファイルの表示
@@ -0,0 +1,32 @@
class PostVersionRecorder < VersionRecorder
def self.record! post:, event_type:, created_by_user:
new(post:, event_type:, created_by_user:).record!
end
def initialize post:, event_type:, created_by_user:
super(record: post, event_type:, created_by_user:)
end
def self.ensure_snapshot! post, created_by_user:
return if post.post_versions.exists?
record!(post:, event_type: :create, created_by_user:)
end
private
def version_class = PostVersion
def version_association = :post_versions
def record_key = :post
def snapshot_attributes
{ title: @record.title,
url: @record.url,
thumbnail_base: @record.thumbnail_base,
video_ms: @record.video_ms,
tags: @record.snapshot_tag_names.join(' '),
parent_post_ids: @record.snapshot_parent_post_ids.join(' '),
original_created_from: @record.original_created_from,
original_created_before: @record.original_created_before }
end
end
+21
ファイルの表示
@@ -0,0 +1,21 @@
module Preview
class HtmlMetadataExtractor
IMAGE_SELECTORS = [
'meta[property="og:image"]',
'meta[name="twitter:image"]',
'meta[name="thumbnail"]'
].freeze
def self.extract(response)
document = Nokogiri::HTML.parse(response.body)
image_url = IMAGE_SELECTORS.filter_map {
document.at_css(_1)&.[]('content')&.strip.presence
}.first
{ title: document.at_css('title')&.text&.strip,
image_url: image_url ? URI.join(response.url, image_url).to_s : nil }
rescue URI::InvalidURIError
{ title: document.at_css('title')&.text&.strip, image_url: nil }
end
end
end
+123
ファイルの表示
@@ -0,0 +1,123 @@
require 'net/http'
require 'json'
module Preview
class HttpFetcher
class FetchFailed < StandardError; end
class FetchTimeout < FetchFailed; end
class ResponseTooLarge < FetchFailed; end
MAX_REDIRECTS = 5
DEFAULT_MAX_BYTES = 5.megabytes
Response = Data.define(:body, :content_type, :url)
def self.fetch(raw_url, max_bytes: DEFAULT_MAX_BYTES, redirects: MAX_REDIRECTS)
uri, addresses = UrlSafety.validate(raw_url)
response = request(uri, addresses.first, max_bytes)
if response.is_a?(Net::HTTPRedirection)
location = response['location']
if redirects.zero?
log_failure(:redirect_limit,
url: uri.to_s,
redirects:,
location:,
content_type: response['content-type'],
content_length: response['content-length'])
raise FetchFailed, 'redirect が多すぎます.'
end
if location.blank?
log_failure(:blank_redirect_location,
url: uri.to_s,
redirects:,
content_type: response['content-type'],
content_length: response['content-length'])
raise FetchFailed, 'redirect 先が不正です.'
end
redirect_url =
begin
URI.join(uri, location).to_s
rescue URI::InvalidURIError => e
log_failure(:invalid_redirect_location,
url: uri.to_s,
redirects:,
location:,
error: e.class.name,
message: e.message)
raise FetchFailed, 'redirect 先が不正です.'
end
return fetch(redirect_url, max_bytes:, redirects: redirects - 1)
end
unless response.is_a?(Net::HTTPSuccess)
log_failure(:http_status,
url: uri.to_s,
code: response.code,
content_type: response['content-type'],
content_length: response['content-length'])
raise FetchFailed, "外部サーバーが HTTP #{ response.code } を返しました."
end
Response.new(response.body, response['content-type'].to_s, uri.to_s)
rescue Net::OpenTimeout, Net::ReadTimeout, Timeout::Error => e
log_failure(:timeout, url: uri&.to_s || raw_url, error: e.class.name, message: e.message)
raise FetchTimeout, e.message
rescue SocketError, SystemCallError, OpenSSL::SSL::SSLError, EOFError => e
log_failure(:network_error, url: uri&.to_s || raw_url, error: e.class.name,
message: e.message)
raise FetchFailed, e.message
end
def self.request(uri, ip_address, max_bytes)
http = Net::HTTP.new(uri.host, uri.port)
http.ipaddr = ip_address
http.use_ssl = uri.scheme == 'https'
http.open_timeout = 5
http.read_timeout = 8
http.write_timeout = 5
request = Net::HTTP::Get.new(uri)
request['User-Agent'] = 'BTRC-Hub thumbnail preview'
request['Accept'] = 'text/html,image/*;q=0.9,*/*;q=0.1'
http.request(request) do |response|
length = response['content-length'].to_i
if length > max_bytes
log_failure(:response_too_large,
url: uri.to_s,
content_type: response['content-type'],
content_length: response['content-length'],
max_bytes:)
raise ResponseTooLarge, '外部データが大きすぎます.'
end
body = +''
response.read_body do |chunk|
body << chunk
next unless body.bytesize > max_bytes
log_failure(:response_too_large,
url: uri.to_s,
content_type: response['content-type'],
content_length: response['content-length'],
bytes_read: body.bytesize,
max_bytes:)
raise ResponseTooLarge, '外部データが大きすぎます.'
end
response.instance_variable_set(:@body, body)
response.instance_variable_set(:@read, true)
return response
end
end
def self.log_failure(reason, **payload)
Rails.logger.warn("preview_http_fetcher_failure #{ { reason:, **payload }.to_json }")
end
private_class_method :request, :log_failure
end
end
+34
ファイルの表示
@@ -0,0 +1,34 @@
module Preview
class KnownSiteExtractor
def self.thumbnail_url(uri)
youtube_thumbnail(uri)
end
def self.youtube_video_id(uri)
case uri.host&.downcase
when 'youtu.be'
uri.path.split('/').reject(&:blank?).first
when 'www.youtube.com', 'youtube.com', 'm.youtube.com'
uri.path == '/watch' ? URI.decode_www_form(uri.query.to_s).to_h['v'] : nil
end&.then { _1 if _1.match?(/\A[A-Za-z0-9_-]{6,20}\z/) }
end
def self.niconico_video_id(uri)
case uri.host&.downcase
when 'www.nicovideo.jp', 'nicovideo.jp'
uri.path[%r{\A/watch/(sm\d+)\z}, 1]
when 'nico.ms'
uri.path[%r{\A/(sm\d+)\z}, 1]
end
end
def self.youtube_thumbnail(uri)
id = youtube_video_id(uri)
return unless id
"https://i.ytimg.com/vi/#{ id }/hqdefault.jpg"
end
private_class_method :youtube_thumbnail
end
end
+102
ファイルの表示
@@ -0,0 +1,102 @@
module Preview
class ThumbnailFetcher
class GenerationFailed < StandardError; end
RASTER_IMAGE_CONTENT_TYPES = [
'image/jpeg', 'image/png', 'image/gif', 'image/webp'
].freeze
HTML_MAX_BYTES = 1.megabyte
NICONICO_XML_MAX_BYTES = 256.kilobytes
def self.fetch(raw_url)
uri, = UrlSafety.validate(raw_url)
known_url = KnownSiteExtractor.thumbnail_url(uri)
image = fetch_image_or_nil(known_url) if known_url
return image if image
niconico_url = niconico_thumbnail_url(uri)
image = fetch_image_or_nil(niconico_url) if niconico_url
return image if image
page = HttpFetcher.fetch(uri.to_s, max_bytes: HTML_MAX_BYTES)
metadata = HtmlMetadataExtractor.extract(page)
raise GenerationFailed, 'サムネール画像が見つかりませんでした.' if metadata[:image_url].blank?
fetch_image!(metadata[:image_url])
end
def self.fetch_image_response(raw_url)
uri, = UrlSafety.validate(raw_url)
response = HttpFetcher.fetch(uri.to_s)
unless Post.remote_thumbnail_image_bytes?(
response.body,
content_type: response.content_type)
raise GenerationFailed, 'サムネール画像が見つかりませんでした.'
end
response
rescue HttpFetcher::FetchTimeout
raise
rescue HttpFetcher::ResponseTooLarge
raise
rescue HttpFetcher::FetchFailed
raise GenerationFailed, 'サムネール画像を取得できませんでした.'
end
def self.title(raw_url)
uri, = UrlSafety.validate(raw_url)
HtmlMetadataExtractor.extract(
HttpFetcher.fetch(uri.to_s, max_bytes: HTML_MAX_BYTES))[:title]
end
def self.fetch_image_or_nil(url)
return nil if url.blank?
response = HttpFetcher.fetch(url)
return nil unless Post.remote_thumbnail_image_bytes?(
response.body,
content_type: response.content_type)
response.body
rescue HttpFetcher::FetchTimeout
raise
rescue HttpFetcher::FetchFailed
nil
end
def self.fetch_image!(url)
fetch_image_response(url).body
end
def self.niconico_thumbnail_url(uri)
video_id = KnownSiteExtractor.niconico_video_id(uri)
return nil if video_id.blank?
response = HttpFetcher.fetch("https://ext.nicovideo.jp/api/getthumbinfo/#{ video_id }",
max_bytes: NICONICO_XML_MAX_BYTES)
xml = Nokogiri::XML(response.body)
return nil unless xml.at_xpath('/nicovideo_thumb_response/@status')&.value == 'ok'
xml.at_xpath('//thumbnail_url')&.text&.strip.presence
rescue HttpFetcher::FetchFailed, HttpFetcher::FetchTimeout => e
payload = {
url: uri.to_s,
video_id:,
error: e.class.name,
message: e.message }
Rails.logger.info("preview_niconico_getthumbinfo_fallback #{ payload.to_json }")
nil
rescue Nokogiri::XML::SyntaxError => e
payload = {
url: uri.to_s,
video_id:,
error: e.class.name,
message: e.message }
Rails.logger.info("preview_niconico_getthumbinfo_fallback #{ payload.to_json }")
nil
end
private_class_method :fetch_image_or_nil, :fetch_image!,
:niconico_thumbnail_url
end
end

変更されたファイルが多すぎるため,一部のファイルは表示されません さらに表示