miteruzo/btrc-hub
1
0
フォーク 0
コード 課題 61 Pull リクエスト 1 リリース Wiki アクティビティ

#399

ソースを参照
このコミットが含まれているのは:
みてるぞ
2026-07-12 12:23:53 +09:00
親 8970edc59f
コミット 7bcb76516c
12個のファイルの変更，1267行の追加，479行の削除
Patchファイルをダウンロード Diffファイルをダウンロード すべてのファイルを展開 すべてのファイルを折り畳む
AGENTS.md
+149
ファイルの表示
@@ -114,6 +114,9 @@ npm run preview
- Ruby: `render` 系メソッド呼び出しでは、keyword 引数付きでも括弧を書かない。 - Ruby: `render` 系メソッド呼び出しでは、keyword 引数付きでも括弧を書かない。
- Ruby: never put a line break immediately before `)`. - Ruby: never put a line break immediately before `)`.
- Ruby: do not use `%w` or `%i`. - Ruby: do not use `%w` or `%i`.
- Ruby: TypeScript / TSX の associative syntax に近い感覚で読むこと。
Hash の `}` は block の `}` ではない。call の `)` も function 定義の
parameter-list `)` ではない。delimiter の役割を見誤らないこと。
- In Ruby, when an `if` condition is split across multiple lines and combines - In Ruby, when an `if` condition is split across multiple lines and combines
clauses with `&&` or `||`, wrap the whole condition in parentheses. clauses with `&&` or `||`, wrap the whole condition in parentheses.
- Ruby hashes are not blocks; keep `}` on the same line as the final pair. - Ruby hashes are not blocks; keep `}` on the same line as the final pair.
@@ -127,6 +130,152 @@ npm run preview
- Keep the first element on the same line as `[` by default. - Keep the first element on the same line as `[` by default.
- If an array would exceed the line limit, break after `[` and indent - If an array would exceed the line limit, break after `[` and indent
elements by 4 spaces. elements by 4 spaces.
### Ruby delimiter and wrapping rules
- Ruby でも、closing delimiter は glyph ではなく syntax role で判定する。
`}` が block close なのか hash close なのか、`)` が method call なのか
grouping なのかを区別してから配置すること。
- Ruby の multi-line method call では、receiver / method と opening `(` を
同じ行に保ち、closing `)` を単独行へ落とさない。
- Ruby の multi-line hash literal / keyword-like argument hash では、opening
`{` を最初の pair と同じ行に置き、closing `}` を最後の pair と同じ行に
置く。Prettier 的な縦開き・縦閉じをしない。
- Ruby の `if` / `unless` / `case` 条件で、複数行に分けるだけで安易に
`if ... end` へ展開しない。局所の既存コードが modifier 形式ならそれに
そろえる。
- Ruby の guard 条件は、1 行で収まるなら modifier 形式を優先する。2 行以上に
なるなら通常の block 形式へ切り替へてよい。
- Ruby の method chain や call argument を折り返す際、call-site の `)`
直前で行を空けたり、closing delimiter を block のやうに独立させない。
Bad:
```rb
source =
if params[:format] == 'google_sheets'
PostImportGoogleSheetsFetcher.fetch!(params[:source],
rate_key: current_user.id)
else
params[:source]
end
parsed = PostImportSourceParser.new(
source:,
format:,
has_header: params[:has_header],
json_path: params[:json_path],
).parse
```
Good:
```rb
source =
if params[:format] == 'google_sheets'
PostImportGoogleSheetsFetcher.fetch!(params[:source],
rate_key: current_user.id)
else
params[:source]
end
parsed = PostImportSourceParser.new(
source:,
format:,
has_header: params[:has_header],
json_path: params[:json_path]).parse
```
Bad:
```rb
result[field] = {
'kind' => kind,
'value' => value['value'].to_s,
'columns' => Array(columns).map { Integer(_1) },
}
```
Good:
```rb
result[field] = {
'kind' => kind,
'value' => value['value'].to_s,
'columns' => Array(columns).map { Integer(_1) } }
```
Bad:
```rb
unless rows.all? { |row|
values = Array(row['values'])
values.length <= columns.length &&
values.all? { |value|
value.is_a?(String) &&
value.bytesize <= PostImportSourceParser::MAX_CELL_BYTES
}
}
raise ArgumentError, '解析結果のセルが不正です．'
end
```
Good:
```rb
raise ArgumentError, '解析結果のセルが不正です．' unless rows.all? { |row|
values = Array(row['values'])
values.length <= columns.length && values.all? { |value|
value.is_a?(String) && value.bytesize <= PostImportSourceParser::MAX_CELL_BYTES
}
}
```
Bad:
```rb
if row['url'].to_s.bytesize > PostImportSourceParser::MAX_CELL_BYTES
raise ArgumentError, 'URL が長すぎます．'
end
```
Good:
```rb
raise ArgumentError, 'URL が長すぎます．' if row['url'].to_s.bytesize > PostImportSourceParser::MAX_CELL_BYTES
```
Bad:
```rb
parameters = row.is_a?(ActionController::Parameters) ? row :
ActionController::Parameters.new(row)
```
Good:
```rb
parameters =
row.is_a?(ActionController::Parameters) ? row : ActionController::Parameters.new(row)
```
Bad:
```rb
PostTagSection.create!(
post_id: post.id,
tag_id:,
begin_ms:,
end_ms:,
)
```
Good:
```rb
PostTagSection.create!(post_id: post.id,
tag_id:,
begin_ms:,
end_ms:)
```
- TypeScript and Python: use GNU-style spacing before parentheses where - TypeScript and Python: use GNU-style spacing before parentheses where
syntactically valid. syntactically valid.
- Never write Ruby, TypeScript, or TSX lines longer than 99 characters. - Never write Ruby, TypeScript, or TSX lines longer than 99 characters.
backend/app/controllers/post_imports_controller.rb
+96 -36
ファイルの表示
@@ -1,16 +1,23 @@
class PostImportsController < ApplicationController class PostImportsController < ApplicationController
MAPPING_PROPERTIES = ['kind', 'value', 'columns'].freeze
MAPPING_KINDS = ['columns', 'fixed'].freeze
before_action :require_member! before_action :require_member!
def parse def parse
source = source =
if params[:format] == 'google_sheets' if params[:format] == 'google_sheets'
PostImportGoogleSheetsFetcher.fetch!(params[:source], rate_key: current_user.id) PostImportGoogleSheetsFetcher.fetch!(params[:source],
rate_key: current_user.id)
else else
params[:source] params[:source]
end end
format = params[:format] == 'google_sheets' ? 'csv' : params[:format] format = params[:format] == 'google_sheets' ? 'csv' : params[:format]
parsed = PostImportSourceParser.new(source:, format:, parsed = PostImportSourceParser.new(
has_header: params[:has_header], json_path: params[:json_path]).parse source:,
format:,
has_header: params[:has_header],
json_path: params[:json_path]).parse
render json: parsed render json: parsed
rescue ArgumentError => e rescue ArgumentError => e
render_bad_request(e.message) render_bad_request(e.message)
@@ -20,10 +27,16 @@ class PostImportsController < ApplicationController
parsed = parsed_params parsed = parsed_params
raw_url_column = params[:url_column].presence || (parsed[:columns].length == 1 ? 0 : nil) raw_url_column = params[:url_column].presence || (parsed[:columns].length == 1 ? 0 : nil)
return render_bad_request('URL 列を選択してください．') if raw_url_column.nil? return render_bad_request('URL 列を選択してください．') if raw_url_column.nil?
url_column = Integer(raw_url_column, exception: false)
return render_bad_request('URL 列が不正です．') if url_column.nil? || url_column.negative? || url_column >= parsed[:columns].length
rows = PostImportPreviewer.new(parsed:, url_column:, mappings: mappings_params(parsed[:columns].length), url_column = Integer(raw_url_column, exception: false)
if url_column.nil? || url_column.negative? || url_column >= parsed[:columns].length
return render_bad_request('URL 列が不正です．')
end
rows = PostImportPreviewer.new(
parsed:,
url_column:,
mappings: mappings_params(parsed[:columns].length),
existing: params[:existing]).preview existing: params[:existing]).preview
render json: parsed.slice(:columns).merge(rows:) render json: parsed.slice(:columns).merge(rows:)
rescue ArgumentError => e rescue ArgumentError => e
@@ -33,17 +46,22 @@ class PostImportsController < ApplicationController
def validate def validate
rows = validate_rows_params rows = validate_rows_params
changed_row = Integer(params[:changed_row], exception: false) changed_row = Integer(params[:changed_row], exception: false)
result = PostImportPreviewer.new(parsed: { columns: [], rows: [] }, url_column: 0, result =
mappings: {}, existing: params[:existing]).preview_rows( PostImportPreviewer.new(
rows:, fetch_metadata: changed_row, parsed: { columns: [], rows: [] },
metadata_cache: { }) url_column: 0,
mappings: {},
existing: params[:existing])
.preview_rows(rows:, fetch_metadata: changed_row, metadata_cache: {})
render json: { rows: result } render json: { rows: result }
rescue ArgumentError => e rescue ArgumentError => e
render_bad_request(e.message) render_bad_request(e.message)
end end
def create def create
result = PostImportRunner.new(actor: current_user, rows: create_rows_params, result = PostImportRunner.new(
actor: current_user,
rows: create_rows_params,
existing: params[:existing]).run existing: params[:existing]).run
render json: result, status: result[:created].positive? ? :created : :ok render json: result, status: result[:created].positive? ? :created : :ok
rescue ArgumentError => e rescue ArgumentError => e
@@ -63,14 +81,20 @@ class PostImportsController < ApplicationController
value = params.require(:parsed).to_unsafe_h.deep_transform_keys { _1.to_s.underscore } value = params.require(:parsed).to_unsafe_h.deep_transform_keys { _1.to_s.underscore }
columns = Array(value['columns']) columns = Array(value['columns'])
rows = Array(value['rows']) rows = Array(value['rows'])
raise ArgumentError, '解析結果の形式が不正です．' unless columns.all?(Hash) && rows.all?(Hash) unless columns.all?(Hash) && rows.all?(Hash)
raise ArgumentError, '解析結果の形式が不正です．'
end
raise ArgumentError, '取込件数が多すぎます．' if rows.length > PostImportSourceParser::MAX_ROWS raise ArgumentError, '取込件数が多すぎます．' if rows.length > PostImportSourceParser::MAX_ROWS
raise ArgumentError, '列数が多すぎます．' if columns.length > 50 raise ArgumentError, '列数が多すぎます．' if columns.length > 50
unless rows.all? { |row| Array(row['values']).length <= columns.length && raise ArgumentError, '解析結果のセルが不正です．' unless rows.all? { |row|
Array(row['values']).all? { _1.is_a?(String) && _1.bytesize <= PostImportSourceParser::MAX_CELL_BYTES } } values = Array(row['values'])
raise ArgumentError, '解析結果のセルが不正です．' values.length <= columns.length && values.all? { |value|
value.is_a?(String) && value.bytesize <= PostImportSourceParser::MAX_CELL_BYTES
}
}
if rows.sum { Array(_1['values']).sum(&:bytesize) } > PostImportSourceParser::MAX_BYTES
raise ArgumentError, '解析結果が大きすぎます．'
end end
raise ArgumentError, '解析結果が大きすぎます．' if rows.sum { Array(_1['values']).sum(&:bytesize) } > PostImportSourceParser::MAX_BYTES
parsed_rows = rows.map do |row| parsed_rows = rows.map do |row|
source_row = Integer(row['source_row'], exception: false) source_row = Integer(row['source_row'], exception: false)
@@ -78,7 +102,9 @@ class PostImportsController < ApplicationController
{ source_row:, values: Array(row['values']) } { source_row:, values: Array(row['values']) }
end end
raise ArgumentError, '元行番号が重複しています．' if parsed_rows.map { _1[:source_row] }.uniq.length != parsed_rows.length if parsed_rows.map { _1[:source_row] }.uniq.length != parsed_rows.length
raise ArgumentError, '元行番号が重複しています．'
end
{ columns: columns.map { |column| column.slice('key', 'label') }, rows: parsed_rows } { columns: columns.map { |column| column.slice('key', 'label') }, rows: parsed_rows }
end end
@@ -92,41 +118,61 @@ class PostImportsController < ApplicationController
unknown_fields = raw.keys - PostImportPreviewer::FIELDS unknown_fields = raw.keys - PostImportPreviewer::FIELDS
raise ArgumentError, '列割当て先が不正です．' if unknown_fields.present? raise ArgumentError, '列割当て先が不正です．' if unknown_fields.present?
permitted = mappings.permit(*PostImportPreviewer::FIELDS.map { |field| permitted =
mappings.permit(*PostImportPreviewer::FIELDS.map { |field|
{ field => [:kind, :value, { columns: [] }] } { field => [:kind, :value, { columns: [] }] }
}).to_h }).to_h
raw.each_with_object({ }) do |(field, value), result| raw.each_with_object({ }) do |(field, value), result|
raise ArgumentError, '列割当ての形式が不正です．' unless value.is_a?(Hash) raise ArgumentError, '列割当ての形式が不正です．' unless value.is_a?(Hash)
raise ArgumentError, '列割当ての項目が不正です．' unless (value.keys - ['kind', 'value', 'columns']).empty? unless (value.keys - MAPPING_PROPERTIES).empty?
raise ArgumentError, '列割当ての項目が不正です．'
end
value = permitted.fetch(field) value = permitted.fetch(field)
kind = value['kind'].presence || 'columns' kind = value['kind'].presence || 'columns'
raise ArgumentError, '列割当ての種別が不正です．' unless kind.in?(['columns', 'fixed']) raise ArgumentError, '列割当ての種別が不正です．' unless kind.in?(MAPPING_KINDS)
columns = value['columns'] columns = value['columns']
unless columns.nil? || (columns.is_a?(Array) && columns.all? { |index| raise ArgumentError, '列番号が不正です．' unless (
columns.nil? || (columns.is_a?(Array) && columns.all? { |index|
parsed = Integer(index, exception: false) parsed = Integer(index, exception: false)
parsed && parsed >= 0 && parsed < column_count parsed && parsed >= 0 && parsed < column_count
}) })
raise ArgumentError, '列番号が不正です．' )
end
raise ArgumentError, '列番号が重複しています．' if Array(columns).uniq.length != Array(columns).length raise ArgumentError, '列番号が重複しています．' if Array(columns).uniq.length != Array(columns).length
raise ArgumentError, '固定値が不正です．' if kind == 'fixed' && !value['value'].is_a?(String) raise ArgumentError, '固定値が不正です．' if kind == 'fixed' && !value['value'].is_a?(String)
result[field] = { 'kind' => kind, 'value' => value['value'].to_s, result[field] = {
'kind' => kind,
'value' => value['value'].to_s,
'columns' => Array(columns).map { Integer(_1) } } 'columns' => Array(columns).map { Integer(_1) } }
end end
end end
def validate_rows_params def validate_rows_params
rows = Array(params[:rows]) rows = params[:rows]
raise ArgumentError, '取込件数が多すぎます．' unless rows.is_a?(Array)
raise ArgumentError, '取込件数が多すぎます．' if rows.length > PostImportSourceParser::MAX_ROWS raise ArgumentError, '取込件数が多すぎます．' if rows.length > PostImportSourceParser::MAX_ROWS
normalised_rows = rows.map do |row| normalised_rows = rows.map do |row|
value = ActionController::Parameters.new(row).permit(:source_row, :sourceRow, :url, :metadata_url, :metadataUrl, unless row.is_a?(Hash) || row.is_a?(ActionController::Parameters)
attributes: {}, provenance: {}, tag_sources: {}, tagSources: {}, raise ArgumentError, '行の形式が不正です．'
fetch_warnings: [], fetchWarnings: [], end
validation_warnings: [], validationWarnings: [])
value = ActionController::Parameters.new(row).permit(
:source_row,
:sourceRow,
:url,
:metadata_url,
:metadataUrl,
attributes: {},
provenance: {},
tag_sources: {},
tagSources: {},
fetch_warnings: [],
fetchWarnings: [],
validation_warnings: [],
validationWarnings: [])
normalised = value.to_h.deep_transform_keys { _1.to_s.underscore } normalised = value.to_h.deep_transform_keys { _1.to_s.underscore }
attributes = normalised.fetch('attributes', { }) attributes = normalised.fetch('attributes', { })
provenance = normalised.fetch('provenance', { }) provenance = normalised.fetch('provenance', { })
@@ -139,16 +185,21 @@ class PostImportsController < ApplicationController
unless provenance.values.all? { _1.in?(['automatic', 'mapped', 'fixed', 'manual']) } unless provenance.values.all? { _1.in?(['automatic', 'mapped', 'fixed', 'manual']) }
raise ArgumentError, '値の由来が不正です．' raise ArgumentError, '値の由来が不正です．'
end end
raise ArgumentError, 'セルが大きすぎます．' if [normalised['url'], *attributes.values].any? { _1.to_s.bytesize > PostImportSourceParser::MAX_CELL_BYTES } raise ArgumentError, 'セルが大きすぎます．' if [normalised['url'], *attributes.values].any? { |value|
value.to_s.bytesize > PostImportSourceParser::MAX_CELL_BYTES
}
if normalised['tag_sources'].present? && if normalised['tag_sources'].present? &&
(!normalised['tag_sources'].is_a?(Hash) || (!normalised['tag_sources'].is_a?(Hash) ||
(normalised['tag_sources'].keys - ['automatic', 'mapped', 'fixed', 'manual']).present?) (normalised['tag_sources'].keys - ['automatic', 'mapped', 'fixed', 'manual']).present?)
raise ArgumentError, 'タグ由来の形式が不正です．' raise ArgumentError, 'タグ由来の形式が不正です．'
end end
if normalised['tag_sources'].present? && if (normalised['tag_sources'].present? &&
(!normalised['tag_sources'].values.all?(String) || (!normalised['tag_sources'].values.all?(String) ||
normalised['tag_sources'].values.any? { _1.bytesize > PostImportSourceParser::MAX_CELL_BYTES } || normalised['tag_sources'].values.any? { |value|
normalised['tag_sources'].values.sum(&:bytesize) > PostImportSourceParser::MAX_CELL_BYTES) value.bytesize > PostImportSourceParser::MAX_CELL_BYTES
} ||
(normalised['tag_sources'].values.sum(&:bytesize) >
PostImportSourceParser::MAX_CELL_BYTES)))
raise ArgumentError, 'タグ由来が大きすぎます．' raise ArgumentError, 'タグ由来が大きすぎます．'
end end
@@ -175,9 +226,18 @@ class PostImportsController < ApplicationController
raise ArgumentError, '取込行の形式が不正です．' raise ArgumentError, '取込行の形式が不正です．'
end end
parameters = row.is_a?(ActionController::Parameters) ? row : ActionController::Parameters.new(row) parameters =
parameters.permit(:source_row, :sourceRow, :url, :metadata_url, :metadataUrl, row.is_a?(ActionController::Parameters) ? row : ActionController::Parameters.new(row)
attributes: {}, provenance: {}, tag_sources: {}, tagSources: {}).to_h parameters.permit(
:source_row,
:sourceRow,
:url,
:metadata_url,
:metadataUrl,
attributes: {},
provenance: {},
tag_sources: {},
tagSources: {}).to_h
end end
end end
end end
backend/app/services/post_creator.rb
+8 -2
ファイルの表示
@@ -90,9 +90,15 @@ class PostCreator
end end
PostTagSection.where(post_id: post.id).destroy_all PostTagSection.where(post_id: post.id).destroy_all
sections.each do |tag_id, ranges| sections.each do |tag_id, ranges|
ranges.each { |begin_ms, end_ms| PostTagSection.create!(post_id: post.id, tag_id:, begin_ms:, end_ms:) } ranges.each do |begin_ms, end_ms|
PostTagSection.create!(post_id: post.id,
tag_id:,
begin_ms:,
end_ms:)
end end
PostTag.where(post_id: post.id, tag_id: (current_ids - desired_ids).to_a).kept.find_each do |post_tag| end
PostTag.where(post_id: post.id,
tag_id: (current_ids - desired_ids).to_a).kept.find_each do |post_tag|
post_tag.discard_by!(@actor) post_tag.discard_by!(@actor)
end end
end end
backend/app/services/post_import_google_sheets_fetcher.rb
+9 -3
ファイルの表示
@@ -17,7 +17,8 @@ class PostImportGoogleSheetsFetcher
def fetch! def fetch!
publication_id, gid = parse! publication_id, gid = parse!
throttle! throttle!
csv_url = "https://#{ HOST }/spreadsheets/d/e/#{ publication_id }/pub?gid=#{ gid }&single=true&output=csv" csv_url = "https://#{ HOST }/spreadsheets/d/e/#{ publication_id }/pub" \
"?gid=#{ gid }&single=true&output=csv"
response = Preview::HttpFetcher.fetch(csv_url, response = Preview::HttpFetcher.fetch(csv_url,
max_bytes: PostImportSourceParser::MAX_BYTES, max_bytes: PostImportSourceParser::MAX_BYTES,
allowed_hosts: [HOST]) allowed_hosts: [HOST])
@@ -26,7 +27,10 @@ class PostImportGoogleSheetsFetcher
response.body response.body
rescue Preview::UrlSafety::UnsafeUrl, Preview::HttpFetcher::FetchFailed, rescue Preview::UrlSafety::UnsafeUrl, Preview::HttpFetcher::FetchFailed,
Preview::HttpFetcher::ResponseTooLarge => e Preview::HttpFetcher::ResponseTooLarge => e
Rails.logger.info("post_import_google_sheets_fetch_failure #{ { error: e.class.name, message: e.message }.to_json }") Rails.logger.info(
'post_import_google_sheets_fetch_failure '\
"#{ { error: e.class.name, message: e.message }.to_json }",
)
raise ArgumentError, 'スプレッドシートを取得できませんでした．ウェブ公開 URL を確認するか、ファイル保存又はコピー貼付けを使用してください．' raise ArgumentError, 'スプレッドシートを取得できませんでした．ウェブ公開 URL を確認するか、ファイル保存又はコピー貼付けを使用してください．'
end end
@@ -51,7 +55,9 @@ class PostImportGoogleSheetsFetcher
def throttle! def throttle!
key = "post-import-google-sheets:#{ @rate_key }" key = "post-import-google-sheets:#{ @rate_key }"
count = Rails.cache.increment(key, 1, expires_in: RATE_WINDOW, initial: 0) count = Rails.cache.increment(key, 1, expires_in: RATE_WINDOW, initial: 0)
raise ArgumentError, 'スプレッドシート取得の回数が多すぎます．しばらくしてから再試行してください．' if count > RATE_LIMIT if count > RATE_LIMIT
raise ArgumentError, 'スプレッドシート取得の回数が多すぎます．しばらくしてから再試行してください．'
end
end end
def csv? response def csv? response
backend/app/services/post_import_previewer.rb
+73 -21
ファイルの表示
@@ -1,5 +1,13 @@
class PostImportPreviewer class PostImportPreviewer
FIELDS = %w[title thumbnail_base original_created_from original_created_before duration tags parent_post_ids].freeze FIELDS = [
'title',
'thumbnail_base',
'original_created_from',
'original_created_before',
'duration',
'tags',
'parent_post_ids',
].freeze
def initialize parsed:, url_column:, mappings: { }, existing: 'skip' def initialize parsed:, url_column:, mappings: { }, existing: 'skip'
@parsed = parsed @parsed = parsed
@@ -17,20 +25,32 @@ class PostImportPreviewer
rows.map do |row| rows.map do |row|
row = row.symbolize_keys row = row.symbolize_keys
values = row[:values] || [] values = row[:values] || []
attributes = row[:attributes]&.stringify_keys || FIELDS.to_h { |field| [field, mapped_value(field, values)] } attributes = row[:attributes]&.stringify_keys || FIELDS.to_h { |field|
[field, mapped_value(field, values)]
}
provenance = row[:provenance]&.stringify_keys || mapped_provenance(attributes) provenance = row[:provenance]&.stringify_keys || mapped_provenance(attributes)
tag_sources = row[:tag_sources]&.stringify_keys || initial_tag_sources(attributes, provenance) tag_sources =
row[:tag_sources]&.stringify_keys || initial_tag_sources(attributes, provenance)
tag_sources['manual'] = attributes['tags'].to_s if provenance['tags'] == 'manual' tag_sources['manual'] = attributes['tags'].to_s if provenance['tags'] == 'manual'
url = row[:url].presence || values[@url_column].to_s.strip url = row[:url].presence || values[@url_column].to_s.strip
provenance['url'] ||= row[:url].present? ? 'manual' : 'mapped' provenance['url'] ||= row[:url].present? ? 'manual' : 'mapped'
url_for_metadata = normalised_url(url) || url url_for_metadata = normalised_url(url) || url
existing_post = normalised_url(url).present? && Post.exists?(url: normalised_url(url)) existing_post = normalised_url(url).present? && Post.exists?(url: normalised_url(url))
if existing_post && @existing == 'skip' if existing_post && @existing == 'skip'
next { source_row: row[:source_row], url: normalised_url(url) || url, attributes:, next {
provenance:, tag_sources:, metadata_url: url_for_metadata, source_row: row[:source_row],
fetch_warnings: [], validation_warnings: ['既存投稿のためスキップします．'], url: normalised_url(url) || url,
warnings: ['既存投稿のためスキップします．'], validation_errors: { }, attributes:,
status: 'warning', existing: true } provenance:,
tag_sources:,
metadata_url: url_for_metadata,
fetch_warnings: [],
validation_warnings: ['既存投稿のためスキップします．'],
warnings: ['既存投稿のためスキップします．'],
validation_errors: { },
status: 'warning',
existing: true,
}
end end
url_changed = row[:metadata_url].present? && row[:metadata_url] != url_for_metadata url_changed = row[:metadata_url].present? && row[:metadata_url] != url_for_metadata
clear_automatic_values!(attributes, provenance, tag_sources) if url_changed clear_automatic_values!(attributes, provenance, tag_sources) if url_changed
@@ -43,7 +63,8 @@ class PostImportPreviewer
when false, nil then false when false, nil then false
else raise ArgumentError, 'メタデータ取得対象が不正です．' else raise ArgumentError, 'メタデータ取得対象が不正です．'
end end
metadata, fetch_warnings = metadata_for(url_for_metadata, should_fetch, metadata_cache, fetch_warnings) metadata, fetch_warnings =
metadata_for(url_for_metadata, should_fetch, metadata_cache, fetch_warnings)
metadata.each do |field, value| metadata.each do |field, value|
if field == 'tags' && provenance[field] != 'manual' if field == 'tags' && provenance[field] != 'manual'
tag_sources['automatic'] = value.to_s tag_sources['automatic'] = value.to_s
@@ -62,19 +83,37 @@ class PostImportPreviewer
errors[:url] = ['URL が重複しています．'] if normal_url.present? && urls.key?(normal_url) errors[:url] = ['URL が重複しています．'] if normal_url.present? && urls.key?(normal_url)
urls[normal_url] = true if normal_url.present? urls[normal_url] = true if normal_url.present?
if normal_url.present? && existing_post if normal_url.present? && existing_post
@existing == 'error' ? errors[:url] = ['既存投稿です．'] : validation_warnings << '既存投稿のためスキップします．' if @existing == 'error'
errors[:url] = ['既存投稿です．']
else
validation_warnings << '既存投稿のためスキップします．'
end
end end
validate_basic_data(attributes, errors) validate_basic_data(attributes, errors)
validate_preview_tags(merged_tags(tag_sources, provenance['tags']), errors, validation_warnings) validate_preview_tags(
merged_tags(tag_sources, provenance['tags']),
errors,
validation_warnings,
)
validate_parents(attributes['parent_post_ids'], errors) validate_parents(attributes['parent_post_ids'], errors)
attributes.delete('url') attributes.delete('url')
attributes['tags'] = merged_tags(tag_sources, provenance['tags']) attributes['tags'] = merged_tags(tag_sources, provenance['tags'])
{ source_row: row[:source_row], url: normal_url || url, attributes:, provenance:, tag_sources:, {
source_row: row[:source_row],
url: normal_url || url,
attributes:,
provenance:,
tag_sources:,
metadata_url: url_for_metadata, metadata_url: url_for_metadata,
fetch_warnings:, validation_warnings:, warnings: fetch_warnings + validation_warnings, fetch_warnings:,
validation_warnings:,
warnings: fetch_warnings + validation_warnings,
validation_errors: errors, validation_errors: errors,
status: errors.present? ? 'error' : ((fetch_warnings + validation_warnings).present? ? 'warning' : 'ready'), status: errors.present? \
existing: normal_url.present? && Post.exists?(url: normal_url) } ? 'error' \
: ((fetch_warnings + validation_warnings).present? ? 'warning' : 'ready'),
existing: normal_url.present? && Post.exists?(url: normal_url),
}
end end
end end
@@ -95,7 +134,9 @@ class PostImportPreviewer
return '' if mapping.blank? return '' if mapping.blank?
return mapping['value'].to_s if mapping['kind'] == 'fixed' return mapping['value'].to_s if mapping['kind'] == 'fixed'
Array(mapping['columns']).filter_map { |index| values[index.to_i].to_s.presence }.join(field == 'tags' ? ' ' : '') Array(mapping['columns']).filter_map { |index|
values[index.to_i].to_s.presence
}.join(field == 'tags' ? ' ' : '')
end end
def mapped_provenance attributes def mapped_provenance attributes
@@ -122,11 +163,14 @@ class PostImportPreviewer
def merged_tags sources, origin = nil def merged_tags sources, origin = nil
return sources['manual'].to_s if origin == 'manual' return sources['manual'].to_s if origin == 'manual'
%w[automatic mapped fixed manual].flat_map { sources[_1].to_s.split }.uniq.join(' ') ['automatic', 'mapped', 'fixed', 'manual'].flat_map {
sources[_1].to_s.split
}.uniq.join(' ')
end end
def clear_automatic_values! attributes, provenance, tag_sources def clear_automatic_values! attributes, provenance, tag_sources
%w[title thumbnail_base original_created_from original_created_before duration].each do |field| ['title', 'thumbnail_base', 'original_created_from',
'original_created_before', 'duration'].each do |field|
attributes[field] = '' if provenance[field] == 'automatic' attributes[field] = '' if provenance[field] == 'automatic'
end end
tag_sources['automatic'] = '' tag_sources['automatic'] = ''
@@ -140,8 +184,13 @@ class PostImportPreviewer
warnings << 'タイトルを取得できませんでした．' if data['title'].blank? warnings << 'タイトルを取得できませんでした．' if data['title'].blank?
warnings << 'サムネールを取得できませんでした．' if data['thumbnail_base'].blank? warnings << 'サムネールを取得できませんでした．' if data['thumbnail_base'].blank?
[data.compact, warnings] [data.compact, warnings]
rescue Preview::UrlSafety::UnsafeUrl, Preview::HttpFetcher::FetchFailed, Preview::HttpFetcher::ResponseTooLarge => e rescue Preview::UrlSafety::UnsafeUrl,
Rails.logger.info("post_import_metadata_fetch_failure #{ { error: e.class.name, message: e.message }.to_json }") Preview::HttpFetcher::FetchFailed,
Preview::HttpFetcher::ResponseTooLarge => e
Rails.logger.info(
"post_import_metadata_fetch_failure "\
"#{ { error: e.class.name, message: e.message }.to_json }",
)
[{ }, ['メタデータを取得できませんでした．']] [{ }, ['メタデータを取得できませんでした．']]
end end
@@ -190,7 +239,10 @@ class PostImportPreviewer
def validate_parents raw, errors def validate_parents raw, errors
ids = raw.to_s.split.map { Integer(_1, exception: false) } ids = raw.to_s.split.map { Integer(_1, exception: false) }
return if ids.compact.empty? && raw.to_s.blank? return if ids.compact.empty? && raw.to_s.blank?
errors[:parent_post_ids] = ['親投稿 Id. が不正です．'] and return if ids.any? { _1.nil? || _1 <= 0 } errors[:parent_post_ids] = ['親投稿 Id. が不正です．'] and return if ids.any? { _1.nil? || _1 <= 0 }
errors[:parent_post_ids] = ['存在しない親投稿 Id. があります．'] if Post.where(id: ids).count != ids.uniq.length if Post.where(id: ids).count != ids.uniq.length
errors[:parent_post_ids] = ['存在しない親投稿 Id. があります．']
end
end end
end end
backend/app/services/post_import_runner.rb
+73 -25
ファイルの表示
@@ -1,6 +1,16 @@
class PostImportRunner class PostImportRunner
MAX_ROWS = PostImportSourceParser::MAX_ROWS MAX_ROWS = PostImportSourceParser::MAX_ROWS
ATTRIBUTE_KEYS = %w[title thumbnail_base original_created_from original_created_before duration tags parent_post_ids video_ms].freeze ATTRIBUTE_KEYS = [
'title',
'thumbnail_base',
'original_created_from',
'original_created_before',
'duration',
'tags',
'parent_post_ids',
'video_ms',
].freeze
def initialize actor:, rows:, existing: 'skip' def initialize actor:, rows:, existing: 'skip'
@actor = actor @actor = actor
@rows = rows @rows = rows
@@ -9,15 +19,21 @@ class PostImportRunner
def run def run
raise ArgumentError, "取込件数は #{ MAX_ROWS } 件までです．" if @rows.length > MAX_ROWS raise ArgumentError, "取込件数は #{ MAX_ROWS } 件までです．" if @rows.length > MAX_ROWS
raise ArgumentError, '取込データが大きすぎます．' if @rows.sum { |row| row.to_json.bytesize } > PostImportSourceParser::MAX_BYTES if @rows.sum { |row| row.to_json.bytesize } > PostImportSourceParser::MAX_BYTES
raise ArgumentError, '取込データが大きすぎます．'
end
normalised_rows = @rows.map { |row| normalise_row(row) } normalised_rows = @rows.map { |row| normalise_row(row) }
source_rows = normalised_rows.map { _1['source_row'] } source_rows = normalised_rows.map { _1['source_row'] }
raise ArgumentError, '元行番号が重複しています．' if source_rows.uniq.length != source_rows.length raise ArgumentError, '元行番号が重複しています．' if source_rows.uniq.length != source_rows.length
results = normalised_rows.map { |row| run_row(row) } results = normalised_rows.map { |row| run_row(row) }
{ created: results.count { _1[:status] == 'created' }, skipped: results.count { _1[:status] == 'skipped' }, {
failed: results.count { _1[:status] == 'failed' }, rows: results } created: results.count { _1[:status] == 'created' },
skipped: results.count { _1[:status] == 'skipped' },
failed: results.count { _1[:status] == 'failed' },
rows: results,
}
end end
private private
@@ -26,20 +42,40 @@ class PostImportRunner
validate_row_structure!(row) validate_row_structure!(row)
attributes = row.fetch('attributes', { }).to_h.transform_keys { _1.to_s.underscore } attributes = row.fetch('attributes', { }).to_h.transform_keys { _1.to_s.underscore }
raise ArgumentError, '取込項目が不正です．' unless (attributes.keys - ATTRIBUTE_KEYS).empty? raise ArgumentError, '取込項目が不正です．' unless (attributes.keys - ATTRIBUTE_KEYS).empty?
raise ArgumentError, '取込項目が大きすぎます．' if attributes.values.any? { _1.to_s.bytesize > PostImportSourceParser::MAX_CELL_BYTES } if attributes.values.any? { _1.to_s.bytesize > PostImportSourceParser::MAX_CELL_BYTES }
raise ArgumentError, 'URL が長すぎます．' if row['url'].to_s.bytesize > PostImportSourceParser::MAX_CELL_BYTES raise ArgumentError, '取込項目が大きすぎます．'
validate_tag_sources!(row['tag_sources'])
preview = PostImportPreviewer.new(parsed: { columns: [], rows: [] }, url_column: 0,
mappings: {}, existing: @existing).preview_rows(
rows: [{ source_row: row['source_row'], url: row['url'],
attributes:, provenance: row['provenance'],
tag_sources: row['tag_sources'] }],
fetch_metadata: false).first
if preview[:validation_errors].present?
return { source_row: row['source_row'], status: 'failed',
errors: preview[:validation_errors] }
end end
return row.slice('source_row').merge(status: 'skipped') if @existing == 'skip' && preview[:existing] if row['url'].to_s.bytesize > PostImportSourceParser::MAX_CELL_BYTES
raise ArgumentError, 'URL が長すぎます．'
end
validate_tag_sources!(row['tag_sources'])
preview = PostImportPreviewer.new(
parsed: { columns: [], rows: [] },
url_column: 0,
mappings: {},
existing: @existing,
).preview_rows(
rows: [{
source_row: row['source_row'],
url: row['url'],
attributes:,
provenance: row['provenance'],
tag_sources: row['tag_sources'],
}],
fetch_metadata: false,
).first
if preview[:validation_errors].present?
return {
source_row: row['source_row'],
status: 'failed',
errors: preview[:validation_errors],
}
end
if @existing == 'skip' && preview[:existing]
return row.slice('source_row').merge(status: 'skipped')
end
attributes['tags'] = preview[:attributes]['tags'] attributes['tags'] = preview[:attributes]['tags']
attributes['url'] = row['url'] attributes['url'] = row['url']
post = PostCreator.new(actor: @actor, attributes:).create! post = PostCreator.new(actor: @actor, attributes:).create!
@@ -53,7 +89,9 @@ class PostImportRunner
rescue ArgumentError, PostCreator::VideoMsParseError rescue ArgumentError, PostCreator::VideoMsParseError
{ source_row: row['source_row'], status: 'failed', errors: { base: ['入力値が不正です．'] } } { source_row: row['source_row'], status: 'failed', errors: { base: ['入力値が不正です．'] } }
rescue StandardError => e rescue StandardError => e
Rails.logger.error("post_import_runner_failure #{ { error: e.class.name, message: e.message }.to_json }") Rails.logger.error(
"post_import_runner_failure #{ { error: e.class.name, message: e.message }.to_json }",
)
{ source_row: row['source_row'], status: 'failed', errors: { base: ['登録中にエラーが発生しました．'] } } { source_row: row['source_row'], status: 'failed', errors: { base: ['登録中にエラーが発生しました．'] } }
end end
@@ -61,9 +99,11 @@ class PostImportRunner
return if sources.blank? return if sources.blank?
sources = sources.to_h sources = sources.to_h
raise ArgumentError unless (sources.keys - %w[automatic mapped fixed manual]).empty? raise ArgumentError unless (sources.keys - ['automatic', 'mapped', 'fixed', 'manual']).empty?
raise ArgumentError unless sources.values.all?(String) raise ArgumentError unless sources.values.all?(String)
raise ArgumentError if sources.values.any? { _1.bytesize > PostImportSourceParser::MAX_CELL_BYTES } if sources.values.any? { _1.bytesize > PostImportSourceParser::MAX_CELL_BYTES }
raise ArgumentError
end
raise ArgumentError if sources.values.sum(&:bytesize) > PostImportSourceParser::MAX_CELL_BYTES raise ArgumentError if sources.values.sum(&:bytesize) > PostImportSourceParser::MAX_CELL_BYTES
end end
@@ -71,12 +111,22 @@ class PostImportRunner
raise ArgumentError unless row['source_row'].is_a?(Integer) && row['source_row'].positive? raise ArgumentError unless row['source_row'].is_a?(Integer) && row['source_row'].positive?
raise ArgumentError unless row['url'].is_a?(String) raise ArgumentError unless row['url'].is_a?(String)
raise ArgumentError unless row['attributes'].is_a?(Hash) raise ArgumentError unless row['attributes'].is_a?(Hash)
raise ArgumentError unless row['attributes'].values.all? { _1.is_a?(String) || _1.is_a?(Numeric) || _1 == true || _1 == false || _1.nil? } unless row['attributes'].values.all? { |value|
value.is_a?(String) ||
value.is_a?(Numeric) ||
value == true ||
value == false ||
value.nil?
}
raise ArgumentError
end
provenance = row['provenance'] provenance = row['provenance']
raise ArgumentError unless provenance.is_a?(Hash) raise ArgumentError unless provenance.is_a?(Hash)
allowed = ATTRIBUTE_KEYS + ['url'] allowed = ATTRIBUTE_KEYS + ['url']
raise ArgumentError unless (provenance.keys - allowed).empty? raise ArgumentError unless (provenance.keys - allowed).empty?
raise ArgumentError unless provenance.values.all? { _1.in?(['automatic', 'mapped', 'fixed', 'manual']) } unless provenance.values.all? { _1.in?(['automatic', 'mapped', 'fixed', 'manual']) }
raise ArgumentError
end
metadata_url = row['metadata_url'] metadata_url = row['metadata_url']
raise ArgumentError unless metadata_url.nil? || metadata_url.is_a?(String) raise ArgumentError unless metadata_url.nil? || metadata_url.is_a?(String)
raise ArgumentError if metadata_url.to_s.bytesize > PostImportSourceParser::MAX_CELL_BYTES raise ArgumentError if metadata_url.to_s.bytesize > PostImportSourceParser::MAX_CELL_BYTES
@@ -86,9 +136,7 @@ class PostImportRunner
end end
def normalise_row row def normalise_row row
unless row.is_a?(Hash) raise ArgumentError, '取込行の形式が不正です．' unless row.is_a?(Hash)
raise ArgumentError, '取込行の形式が不正です．'
end
normalised = row.deep_transform_keys { _1.to_s.underscore } normalised = row.deep_transform_keys { _1.to_s.underscore }
source_row = Integer(normalised['source_row'], exception: false) source_row = Integer(normalised['source_row'], exception: false)
backend/app/services/post_import_source_parser.rb
+27 -9
ファイルの表示
@@ -30,9 +30,17 @@ class PostImportSourceParser
ensure_limits!(table) ensure_limits!(table)
width = table.map(&:length).max.to_i width = table.map(&:length).max.to_i
raise ArgumentError, "列数は #{ MAX_COLUMNS } 列までです．" if width > MAX_COLUMNS raise ArgumentError, "列数は #{ MAX_COLUMNS } 列までです．" if width > MAX_COLUMNS
{ columns: (0...width).map { |index| { key: index.to_s, {
label: "#{ column_name(index) }：#{ headers&.[](index).presence || '列' }" } }, columns: (0...width).map { |index|
rows: table.each_with_index.map { |values, index| { source_row: index + (@has_header ? 2 : 1), values: } } } {
key: index.to_s,
label: "#{ column_name(index) }：#{ headers&.[](index).presence || '列' }",
}
},
rows: table.each_with_index.map { |values, index|
{ source_row: index + (@has_header ? 2 : 1), values: }
},
}
rescue CSV::MalformedCSVError => e rescue CSV::MalformedCSVError => e
raise ArgumentError, "CSV・TSV の形式が不正です: #{ e.message }" raise ArgumentError, "CSV・TSV の形式が不正です: #{ e.message }"
end end
@@ -59,17 +67,27 @@ class PostImportSourceParser
ensure_limits!(value) ensure_limits!(value)
unless value.all?(Hash) unless value.all?(Hash)
invalid = value.each_with_index.filter_map { |item, index| index + 1 unless item.is_a?(Hash) } invalid = value.each_with_index.filter_map { |item, index|
index + 1 unless item.is_a?(Hash)
}
raise ArgumentError, "JSON の投稿候補は object の配列にしてください: 行 #{ invalid.join(', ') }" raise ArgumentError, "JSON の投稿候補は object の配列にしてください: 行 #{ invalid.join(', ') }"
end end
keys = value.flat_map(&:keys).map(&:to_s).uniq keys = value.flat_map(&:keys).map(&:to_s).uniq
raise ArgumentError, "列数は #{ MAX_COLUMNS } 列までです．" if keys.length > MAX_COLUMNS raise ArgumentError, "列数は #{ MAX_COLUMNS } 列までです．" if keys.length > MAX_COLUMNS
{ columns: keys.each_with_index.map { |key, index| { key:, label: "#{ column_name(index) }：#{ key }" } }, {
rows: value.each_with_index.map { |item, index| { source_row: index + 1, columns: keys.each_with_index.map { |key, index|
{ key:, label: "#{ column_name(index) }：#{ key }" }
},
rows: value.each_with_index.map { |item, index|
{
source_row: index + 1,
values: keys.map { |key| values: keys.map { |key|
value = item.key?(key) ? item[key] : item[key.to_sym] json_value = item.key?(key) ? item[key] : item[key.to_sym]
json_cell(value) json_cell(json_value)
} } } } },
}
},
}
rescue JSON::ParserError, KeyError, ArgumentError => e rescue JSON::ParserError, KeyError, ArgumentError => e
raise ArgumentError, "JSON の形式が不正です: #{ e.message }" raise ArgumentError, "JSON の形式が不正です: #{ e.message }"
end end
backend/app/services/post_metadata_fetcher.rb
+15 -4
ファイルの表示
@@ -1,10 +1,19 @@
class PostMetadataFetcher class PostMetadataFetcher
def self.fetch raw_url def self.fetch raw_url
uri, = Preview::UrlSafety.validate(raw_url) uri, = Preview::UrlSafety.validate(raw_url)
response = Preview::HttpFetcher.fetch(uri.to_s, max_bytes: Preview::ThumbnailFetcher::HTML_MAX_BYTES) response = Preview::HttpFetcher.fetch(
uri.to_s,
max_bytes: Preview::ThumbnailFetcher::HTML_MAX_BYTES,
)
metadata = Preview::HtmlMetadataExtractor.extract(response) metadata = Preview::HtmlMetadataExtractor.extract(response)
document = Nokogiri::HTML.parse(response.body) document = Nokogiri::HTML.parse(response.body)
content = -> name { document.at_css("meta[property='#{ name }'], meta[name='#{ name }']")&.[]('content')&.strip.presence } content = lambda { |name|
document
.at_css("meta[property='#{ name }'], meta[name='#{ name }']")
&.[]('content')
&.strip
&.presence
}
duration = content.call('og:video:duration') || content.call('video:duration') duration = content.call('og:video:duration') || content.call('video:duration')
published = content.call('article:published_time') || content.call('date') published = content.call('article:published_time') || content.call('date')
created_range = original_created_range(published) created_range = original_created_range(published)
@@ -30,7 +39,8 @@ class PostMetadataFetcher
raw = value.to_s.strip raw = value.to_s.strip
from = Time.zone.parse(raw) from = Time.zone.parse(raw)
zone = '(?:Z|[+-]\d{2}:?\d{2})?' zone = '(?:Z|[+-]\d{2}:?\d{2})?'
before = case raw before =
case raw
when /\A\d{4}\z/ then from + 1.year when /\A\d{4}\z/ then from + 1.year
when /\A\d{4}-\d{2}\z/ then from + 1.month when /\A\d{4}-\d{2}\z/ then from + 1.month
when /\A\d{4}-\d{2}-\d{2}\z/ then from + 1.day when /\A\d{4}-\d{2}-\d{2}\z/ then from + 1.day
@@ -39,7 +49,8 @@ class PostMetadataFetcher
when /\A\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}#{ zone }\z/ then from + 1.second when /\A\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}#{ zone }\z/ then from + 1.second
when /\A\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.(\d+)#{ zone }\z/ when /\A\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.(\d+)#{ zone }\z/
from + (10**(-Regexp.last_match(1).length)) from + (10**(-Regexp.last_match(1).length))
else return nil else
return nil
end end
[from, before] [from, before]
rescue ArgumentError, TypeError rescue ArgumentError, TypeError
backend/app/services/preview/http_fetcher.rb
+4 -1
ファイルの表示
@@ -71,7 +71,10 @@ module Preview
log_failure(:timeout, url: uri&.to_s || raw_url, error: e.class.name, message: e.message) log_failure(:timeout, url: uri&.to_s || raw_url, error: e.class.name, message: e.message)
raise FetchTimeout, e.message raise FetchTimeout, e.message
rescue SocketError, SystemCallError, OpenSSL::SSL::SSLError, EOFError => e rescue SocketError, SystemCallError, OpenSSL::SSL::SSLError, EOFError => e
log_failure(:network_error, url: uri&.to_s || raw_url, error: e.class.name, message: e.message) log_failure(:network_error,
url: uri&.to_s || raw_url,
error: e.class.name,
message: e.message)
raise FetchFailed, e.message raise FetchFailed, e.message
end end
frontend/src/components/TopNav.tsx
+2 -2
ファイルの表示
@@ -26,7 +26,7 @@ export const menuOutline = (
tag?: Tag | null tag?: Tag | null
material?: Material | null material?: Material | null
wikiId: number | null wikiId: number | null
user: User | null, user: User | null
pathName: string }, pathName: string },
): Menu => { ): Menu => {
const postCount = tag?.postCount ?? material?.tag?.postCount ?? 0 const postCount = tag?.postCount ?? material?.tag?.postCount ?? 0
@@ -43,7 +43,7 @@ export const menuOutline = (
{ name: '一覧', to: '/posts' }, { name: '一覧', to: '/posts' },
{ name: '検索', to: '/posts/search' }, { name: '検索', to: '/posts/search' },
{ name: '追加', to: '/posts/new' }, { name: '追加', to: '/posts/new' },
{ name: 'インポート', to: '/posts/import' }, { name: '取込', to: '/posts/import' },
{ name: '全体履歴', to: '/posts/changes' }, { name: '全体履歴', to: '/posts/changes' },
{ name: 'ヘルプ', to: '/wiki/ヘルプ:広場' }] }, { name: 'ヘルプ', to: '/wiki/ヘルプ:広場' }] },
{ name: 'タグ', to: '/tags', subMenu: [ { name: 'タグ', to: '/tags', subMenu: [
frontend/src/pages/posts/PostImportPage.tsx
+595 -173
ファイルの表示
@@ -2,10 +2,10 @@ import { useEffect, useRef, useState } from 'react'
import { Helmet } from 'react-helmet-async' import { Helmet } from 'react-helmet-async'
import FieldError from '@/components/common/FieldError' import FieldError from '@/components/common/FieldError'
import PrefetchLink from '@/components/PrefetchLink'
import Form from '@/components/common/Form' import Form from '@/components/common/Form'
import FormField from '@/components/common/FormField' import FormField from '@/components/common/FormField'
import PageTitle from '@/components/common/PageTitle' import PageTitle from '@/components/common/PageTitle'
import PrefetchLink from '@/components/PrefetchLink'
import MainArea from '@/components/layout/MainArea' import MainArea from '@/components/layout/MainArea'
import { Button } from '@/components/ui/button' import { Button } from '@/components/ui/button'
import { toast } from '@/components/ui/use-toast' import { toast } from '@/components/ui/use-toast'
@@ -20,7 +20,25 @@ import type { FC } from 'react'
import type { User } from '@/types' import type { User } from '@/types'
type Props = { user: User | null } type Props = { user: User | null }
type Column = { key: string, label: string }
type Column = {
key: string
label: string }
type ParsedRow = {
sourceRow: number
values: string[] }
type ParsedImport = {
columns: Column[]
rows: ParsedRow[] }
type Origin = 'automatic' | 'mapped' | 'fixed' | 'manual'
type ImportStatus = 'pending' | 'created' | 'skipped' | 'failed'
type ImportResultStatus = 'created' | 'skipped' | 'failed'
type Phase = 'source' | 'mapping' | 'preview' | 'result'
type ImportRow = { type ImportRow = {
sourceRow: number sourceRow: number
url: string url: string
@@ -28,42 +46,125 @@ type ImportRow = {
warnings: string[] warnings: string[]
validationErrors: Record<string, string[]> validationErrors: Record<string, string[]>
importErrors?: Record<string, string[]> importErrors?: Record<string, string[]>
provenance: Record<string, 'automatic' | 'mapped' | 'fixed' | 'manual'> provenance: Record<string, Origin>
tagSources?: Record<'automatic' | 'mapped' | 'fixed' | 'manual', string> tagSources?: Record<Origin, string>
status: 'ready' | 'warning' | 'error' status: 'ready' | 'warning' | 'error'
existing: boolean existing: boolean
metadataUrl?: string metadataUrl?: string
createdPostId?: number createdPostId?: number
importStatus?: 'pending' | 'created' | 'skipped' | 'failed' } importStatus?: ImportStatus }
type ParsedImport = { columns: Column[], rows: { sourceRow: number, values: string[] }[] }
type Phase = 'source' | 'mapping' | 'preview' | 'result' type ImportResultRow = {
type ImportResultStatus = 'created' | 'skipped' | 'failed' sourceRow: number
status: ImportResultStatus
post?: { id: number }
errors?: Record<string, string[]> }
type Mapping = { columns: string[] }
const MAPPING_FIELDS = [
'title',
'thumbnail_base',
'original_created_from',
'original_created_before',
'duration',
'tags',
'parent_post_ids'] as const
const RESULT_STATUSES: ImportResultStatus[] = ['created', 'skipped', 'failed']
const ORIGIN_LABELS: Record<Origin, string> = {
automatic: '自動取得',
mapped: '入力データ',
fixed: '固定値',
manual: '手修正' }
const detectFormat = (source: string): 'csv' | 'tsv' => const detectFormat = (source: string): 'csv' | 'tsv' =>
source.includes ('\t') ? 'tsv' : (source.includes (',') ? 'csv' : 'tsv') source.includes ('\t')
? 'tsv'
: source.includes (',')
? 'csv'
: 'tsv'
const publicGoogleSheetURL = (source: string): boolean => const publicGoogleSheetURL = (source: string): boolean =>
/^https:\/\/docs\.google\.com\/spreadsheets\/d\/e\/[A-Za-z0-9_-]+\/pubhtml#gid=\d+$/.test ( /^https:\/\/docs\.google\.com\/spreadsheets\/d\/e\/[A-Za-z0-9_-]+\/pubhtml#gid=\d+$/.test (
source.trim ()) source.trim ())
const mergeValidatedRows = (
current: ImportRow[],
validated: ImportRow[],
): ImportRow[] =>
current
.map (previous => {
if (previous.importStatus === 'created')
return previous
const row = validated.find (_1 => _1.sourceRow === previous.sourceRow)
return row
? {
...row,
importStatus: previous.importStatus,
createdPostId: previous.createdPostId,
importErrors: previous.importErrors,
validationErrors: row.validationErrors ?? { } }
: previous
})
.concat (validated.filter (row =>
!(current.some (_1 => _1.sourceRow === row.sourceRow))))
const rowMessages = (row: ImportRow): string[] => [
...row.warnings,
...Object.values (row.validationErrors ?? { }).flat (),
...Object.values (row.importErrors ?? { }).flat ()]
const sampleURLCount = (rows: ParsedRow[], index: number): number =>
rows.filter (row => /^https?:\/\//.test (row.values[index] ?? '')).length
const renderMappingSelect = (
columns: Column[],
field: string,
value: string,
update: (field: string, value: string) => void,
) => (
<label key={field}>
{field}
<select
value={value}
onChange={ev => update (field, ev.target.value)}>
<option value="">指定なし</option>
{columns.map ((column, index) => (
<option key={column.key} value={index}>
{column.label}
</option>))}
</select>
</label>)
const PostImportPage: FC<Props> = ({ user }) => { const PostImportPage: FC<Props> = ({ user }) => {
const editable = canEditContent (user)
const [source, setSource] = useState ('') const [source, setSource] = useState ('')
const [phase, setPhase] = useState<Phase> ('source') const [phase, setPhase] = useState<Phase> ('source')
const [format, setFormat] = useState<'csv' | 'tsv' | 'json' | 'google_sheets'> ('tsv') const [format, setFormat] =
useState<'csv' | 'tsv' | 'json' | 'google_sheets'> ('tsv')
const [hasHeader, setHasHeader] = useState (false) const [hasHeader, setHasHeader] = useState (false)
const [jsonPath, setJsonPath] = useState ('') const [jsonPath, setJsonPath] = useState ('')
const [urlColumn, setUrlColumn] = useState ('') const [urlColumn, setUrlColumn] = useState ('')
const [parsed, setParsed] = useState<ParsedImport | null> (null) const [parsed, setParsed] = useState<ParsedImport | null> (null)
const [columns, setColumns] = useState<Column[]> ([]) const [columns, setColumns] = useState<Column[]> ([])
const [rows, setRows] = useState<ImportRow[]> ([]) const [rows, setRows] = useState<ImportRow[]> ([])
const [results, setResults] = useState<{ sourceRow: number, status: ImportResultStatus, post?: { id: number }, errors?: Record<string, string[]> }[]> ([]) const [results, setResults] = useState<ImportResultRow[]> ([])
const [mappings, setMappings] = useState<Record<string, { columns: string[] }>> ({ }) const [mappings, setMappings] = useState<Record<string, Mapping>> ({ })
const [existing, setExisting] = useState<'skip' | 'error'> ('skip') const [existing, setExisting] = useState<'skip' | 'error'> ('skip')
const [loading, setLoading] = useState (false) const [loading, setLoading] = useState (false)
const validationGeneration = useRef (0) const validationGeneration = useRef (0)
const editable = canEditContent (user)
const parseSource = async () => { const parseSource = async () => {
setLoading (true) setLoading (true)
@@ -80,22 +181,26 @@ const PostImportPage: FC<Props> = ({ user }) => {
setMappings ({ }) setMappings ({ })
setRows ([]) setRows ([])
setResults ([]) setResults ([])
setUrlColumn ('')
if (data.columns.length === 1) if (data.columns.length === 1)
{
setUrlColumn ('0') setUrlColumn ('0')
}
else else
{ {
const candidate = data.columns.map ((_, index) => ({ index, const candidate = data.columns
count: data.rows.filter (row => /^https?:\/\//.test (row.values[index] ?? '')).length })) .map ((_, index) => ({
index,
count: sampleURLCount (data.rows, index) }))
.sort ((a, b) => b.count - a.count)[0] .sort ((a, b) => b.count - a.count)[0]
if (candidate?.count) setUrlColumn (candidate?.count ? String (candidate.index) : '')
setUrlColumn (String (candidate.index))
} }
setPhase ('mapping') setPhase ('mapping')
} }
catch catch
{ {
toast ({ title: '読込みに失敗しました', description: '入力形式を確認してください。' }) toast ({
title: '読込みに失敗しました',
description: '入力形式を確認してください．' })
} }
finally finally
{ {
@@ -106,6 +211,7 @@ const PostImportPage: FC<Props> = ({ user }) => {
const preview = async () => { const preview = async () => {
if (!parsed || urlColumn === '') if (!parsed || urlColumn === '')
return return
setLoading (true) setLoading (true)
try try
{ {
@@ -121,9 +227,12 @@ const PostImportPage: FC<Props> = ({ user }) => {
{ {
const message = const message =
isApiError<{ message?: string, baseErrors?: string[] }> (error) isApiError<{ message?: string, baseErrors?: string[] }> (error)
? error.response?.data?.message ?? error.response?.data?.baseErrors?.[0] ? error.response?.data?.message
?? error.response?.data?.baseErrors?.[0]
: undefined : undefined
toast ({ title: '投稿プレビューに失敗しました', description: message ?? '入力を確認してください。' }) toast ({
title: '投稿プレビューに失敗しました',
description: message ?? '入力を確認してください．' })
} }
finally finally
{ {
@@ -131,124 +240,142 @@ const PostImportPage: FC<Props> = ({ user }) => {
} }
} }
const mergeValidatedRows = (current: ImportRow[], validated: ImportRow[]): ImportRow[] => const validateRows = async (
current.map (previous => { nextRows: ImportRow[],
if (previous.importStatus === 'created') changedRow: number,
return previous ) => {
const row = validated.find (_1 => _1.sourceRow === previous.sourceRow)
return row ? { ...row,
importStatus: previous.importStatus,
createdPostId: previous.createdPostId,
importErrors: previous.importErrors,
validationErrors: row.validationErrors ?? { } } : previous
}).concat (validated.filter (row => !current.some (_1 => _1.sourceRow === row.sourceRow)))
const updateAttribute = async (index: number, field: string, value: string) => {
const row = rows[index]
const next = { ...row,
attributes: { ...row.attributes, [field]: value },
provenance: { ...row.provenance, [field]: 'manual' },
importStatus: 'pending' as const,
importErrors: undefined }
const nextRows = rows.map ((currentRow, rowIndex) => rowIndex === index ? next : currentRow)
setRows (nextRows)
const generation = ++validationGeneration.current const generation = ++validationGeneration.current
setLoading (true) setLoading (true)
try try
{ {
const validated = await apiPost<{ rows: ImportRow[] }> ('/posts/import/validate', const validated = await apiPost<{ rows: ImportRow[] }> ('/posts/import/validate', {
{ rows: nextRows.filter (row => row.importStatus !== 'created'), existing, rows: nextRows.filter (row => row.importStatus !== 'created'),
changed_row: -1 }) existing,
changed_row: changedRow })
if (generation === validationGeneration.current) if (generation === validationGeneration.current)
setRows (current => mergeValidatedRows (current, validated.rows)) setRows (current => mergeValidatedRows (current, validated.rows))
} }
finally
{
if (generation === validationGeneration.current)
setLoading (false)
}
}
const updateAttribute = async (
index: number,
field: string,
value: string,
) => {
const row = rows[index]
const next = {
...row,
attributes: { ...row.attributes, [field]: value },
provenance: { ...row.provenance, [field]: 'manual' },
importStatus: 'pending' as const,
importErrors: undefined }
const nextRows = rows.map ((currentRow, rowIndex) =>
rowIndex === index ? next : currentRow)
setRows (nextRows)
try
{
await validateRows (nextRows, -1)
}
catch catch
{ {
toast ({ title: '行の再検証に失敗しました' }) toast ({ title: '行の再検証に失敗しました' })
} }
finally
{
if (generation === validationGeneration.current)
setLoading (false)
}
} }
const updateURL = async (index: number, value: string) => { const updateURL = async (index: number, value: string) => {
const row = rows[index] const row = rows[index]
const next = { ...row, url: value, provenance: { ...row.provenance, url: 'manual' }, const next = {
importStatus: 'pending' as const, importErrors: undefined } ...row,
const nextRows = rows.map ((currentRow, rowIndex) => rowIndex === index ? next : currentRow) url: value,
provenance: { ...row.provenance, url: 'manual' },
importStatus: 'pending' as const,
importErrors: undefined }
const nextRows = rows.map ((currentRow, rowIndex) =>
rowIndex === index ? next : currentRow)
setRows (nextRows) setRows (nextRows)
const generation = ++validationGeneration.current
setLoading (true)
try try
{ {
const validated = await apiPost<{ rows: ImportRow[] }> ('/posts/import/validate', await validateRows (nextRows, row.sourceRow)
{ rows: nextRows.filter (row => row.importStatus !== 'created'), existing,
changed_row: row.sourceRow })
if (generation === validationGeneration.current)
setRows (current => mergeValidatedRows (current, validated.rows))
} }
catch catch
{ {
toast ({ title: 'URL の再検証に失敗しました' }) toast ({ title: 'URL の再検証に失敗しました' })
} }
finally
{
if (generation === validationGeneration.current)
setLoading (false)
}
} }
const updateExisting = async (value: 'skip' | 'error') => { const updateExisting = async (value: 'skip' | 'error') => {
setExisting (value) setExisting (value)
const pendingRows = rows.map (row => row.importStatus === 'created' ? row : const pendingRows = rows.map (row =>
{ ...row, importStatus: 'pending' as const, importErrors: undefined }) row.importStatus === 'created'
? row
: {
...row,
importStatus: 'pending' as const,
importErrors: undefined })
setRows (pendingRows) setRows (pendingRows)
const generation = ++validationGeneration.current
setLoading (true)
try try
{ {
const validated = await apiPost<{ rows: ImportRow[] }> ('/posts/import/validate', const generation = ++validationGeneration.current
{ rows: pendingRows.filter (row => row.importStatus !== 'created'), existing: value, changed_row: -1 }) setLoading (true)
const validated = await apiPost<{ rows: ImportRow[] }> ('/posts/import/validate', {
rows: pendingRows.filter (row => row.importStatus !== 'created'),
existing: value,
changed_row: -1 })
if (generation === validationGeneration.current) if (generation === validationGeneration.current)
setRows (current => mergeValidatedRows (current, validated.rows)) setRows (current => mergeValidatedRows (current, validated.rows))
} }
finally finally
{ {
if (generation === validationGeneration.current) if (validationGeneration.current > 0)
setLoading (false) setLoading (false)
} }
} }
const submit = async () => { const submit = async () => {
const targets = rows.filter (row => row.importStatus == null || row.importStatus === 'pending') const targets = rows.filter (row =>
row.importStatus == null || row.importStatus === 'pending')
if (targets.length === 0) if (targets.length === 0)
return return
setLoading (true) setLoading (true)
try try
{ {
const result = await apiPost<{ created: number, skipped: number, failed: number, const result = await apiPost<{
rows: { sourceRow: number, status: ImportResultStatus, post?: { id: number }, errors?: Record<string, string[]> }[] }> ('/posts/import', { created: number
rows: targets.map (row => ({ sourceRow: row.sourceRow, skipped: number
failed: number
rows: ImportResultRow[] }> ('/posts/import', {
rows: targets.map (row => ({
sourceRow: row.sourceRow,
url: row.url, url: row.url,
attributes: row.attributes, attributes: row.attributes,
provenance: row.provenance, provenance: row.provenance,
tagSources: row.tagSources, tagSources: row.tagSources,
metadataUrl: row.metadataUrl })), metadataUrl: row.metadataUrl })),
existing }) existing })
if (result.rows.some (row => !(['created', 'skipped', 'failed'] as string[]).includes (row.status))) if (result.rows.some (row => !(RESULT_STATUSES.includes (row.status))))
throw new Error ('不正な登録結果です。') throw new Error ('不正な登録結果です．')
toast ({ title: `登録完了: ${result.created} 件`,
toast ({
title: `登録完了: ${ result.created } 件`,
description: `スキップ ${ result.skipped } 件、失敗 ${ result.failed } 件` }) description: `スキップ ${ result.skipped } 件、失敗 ${ result.failed } 件` })
setPhase ('result') setPhase ('result')
setResults (result.rows) setResults (result.rows)
setRows (current => current.map (row => { setRows (current => current.map (row => {
const resultRow = result.rows.find (_1 => _1.sourceRow === row.sourceRow) const resultRow = result.rows.find (_1 => _1.sourceRow === row.sourceRow)
return resultRow ? { ...row, importStatus: resultRow.status, return resultRow
? {
...row,
importStatus: resultRow.status,
createdPostId: resultRow.post?.id, createdPostId: resultRow.post?.id,
importErrors: resultRow.errors, importErrors: resultRow.errors,
validationErrors: row.validationErrors } : row validationErrors: row.validationErrors }
: row
})) }))
} }
catch catch
@@ -261,142 +388,437 @@ const PostImportPage: FC<Props> = ({ user }) => {
} }
} }
const updateMapping = (field: string, value: string) =>
setMappings (current => ({
...current,
[field]: { columns: value === '' ? [] : [value] } }))
if (!(editable)) if (!(editable))
return <Forbidden/> return <Forbidden/>
return ( return (
<MainArea> <MainArea>
<Helmet><title>{`投稿インポート | ${SITE_TITLE}`}</title></Helmet> <Helmet>
<title>{`投稿インポート | ${ SITE_TITLE }`}</title>
</Helmet>
<PageTitle>投稿インポート</PageTitle> <PageTitle>投稿インポート</PageTitle>
{phase === 'source' ? ( {phase === 'source'
? (
<Form> <Form>
<p className="text-sm text-neutral-600 dark:text-neutral-300"> <p className="text-sm text-neutral-600 dark:text-neutral-300">
URL を一行に一件ずつ貼り付けるか、CSV・TSV・JSON を入力してください。 URL を一行に一件ずつ貼り付けるか、
CSV・TSV・JSON を入力してください．
</p> </p>
{publicGoogleSheetURL (source) && ( {publicGoogleSheetURL (source) && (
<fieldset className="space-y-1 rounded border p-3 dark:border-neutral-700"> <fieldset className="space-y-1 rounded border p-3 dark:border-neutral-700">
<legend>公開 Google スプレッドシート URL</legend> <legend>公開 Google スプレッドシート URL</legend>
<label className="mr-3"><input type="radio" checked={format === 'google_sheets'} <label className="mr-3">
onChange={() => setFormat ('google_sheets')}/>シート内データを読み込む</label> <input
<label><input type="radio" checked={format !== 'google_sheets'} type="radio"
onChange={() => setFormat ('tsv')}/>URL 自体を投稿として扱う</label> checked={format === 'google_sheets'}
onChange={() => setFormat ('google_sheets')}/>
シート内データを読み込む
</label>
<label>
<input
type="radio"
checked={format !== 'google_sheets'}
onChange={() => setFormat ('tsv')}/>
URL 自体を投稿として扱う
</label>
</fieldset>)} </fieldset>)}
<FormField label="入力形式"> <FormField label="入力形式">
{() => <select value={format} onChange={ev => setFormat (ev.target.value as typeof format)} {() => (
<select
value={format}
onChange={ev => setFormat (ev.target.value as typeof format)}
className={inputClass (false)}> className={inputClass (false)}>
<option value="tsv">テキスト・TSV</option><option value="csv">CSV</option> <option value="tsv">テキスト・TSV</option>
<option value="csv">CSV</option>
<option value="json">JSON</option> <option value="json">JSON</option>
<option value="google_sheets">公開 Google スプレッドシート</option> <option value="google_sheets">
</select>} 公開 Google スプレッドシート
</option>
</select>)}
</FormField> </FormField>
<FormField label="入力データ"> <FormField label="入力データ">
{() => <textarea value={source} rows={12} className={inputClass (false)} {() => (
onChange={ev => { setSource (ev.target.value); if (format !== 'json' && format !== 'google_sheets') setFormat (detectFormat (ev.target.value)) }}/>} <textarea
value={source}
rows={12}
className={inputClass (false)}
onChange={ev => {
setSource (ev.target.value)
if (format !== 'json' && format !== 'google_sheets')
setFormat (detectFormat (ev.target.value))
}}/>)}
</FormField> </FormField>
<input type="file" accept=".txt,.csv,.tsv,.json" onChange={ev => { <input
type="file"
accept=".txt,.csv,.tsv,.json"
onChange={ev => {
const file = ev.target.files?.[0] const file = ev.target.files?.[0]
if (file) file.text ().then (text => { setSource (text); setFormat (file.name.endsWith ('.json') ? 'json' : detectFormat (text)) }) if (file)
{
file.text ().then (text => {
setSource (text)
setFormat (
file.name.endsWith ('.json')
? 'json'
: detectFormat (text))
})
}
}}/> }}/>
{format !== 'json' && <label className="flex gap-2"><input type="checkbox" checked={hasHeader} {format !== 'json' && (
onChange={ev => setHasHeader (ev.target.checked)}/>見出し行を使用する</label>} <label className="flex gap-2">
{format === 'json' && <FormField label="投稿候補の位置（例: data.posts）"> <input
{() => <input value={jsonPath} onChange={ev => setJsonPath (ev.target.value)} className={inputClass (false)}/>} type="checkbox"
</FormField>} checked={hasHeader}
{format === 'google_sheets' && <p className="text-sm text-neutral-600 dark:text-neutral-300"> onChange={ev => setHasHeader (ev.target.checked)}/>
非公開シートには対応していません。ウェブ公開 URL を使用するか、CSV を保存して選択、又は内容を貼り付けてください。 見出し行を使用する
</p>} </label>)}
<Button type="button" onClick={parseSource} disabled={loading || !(source)}>列を解析</Button> {format === 'json' && (
</Form>) : phase === 'mapping' && parsed ? ( <FormField label="投稿候補の位置（例: data.posts）">
{() => (
<input
value={jsonPath}
onChange={ev => setJsonPath (ev.target.value)}
className={inputClass (false)}/>)}
</FormField>)}
{format === 'google_sheets' && (
<p className="text-sm text-neutral-600 dark:text-neutral-300">
非公開シートには対応していません．
ウェブ公開 URL を使用するか、CSV を保存して選択、
又は内容を貼り付けてください．
</p>)}
<Button
type="button"
onClick={parseSource}
disabled={loading || !(source)}>
列を解析
</Button>
</Form>)
: phase === 'mapping' && parsed
? (
<Form> <Form>
<PageTitle>列の割当て</PageTitle> <PageTitle>列の割当て</PageTitle>
<p>URL 列と補助列を指定してから、投稿情報を取得します。</p> <p>URL 列と補助列を指定してから、投稿情報を取得します．</p>
<label>URL 列 <select value={urlColumn} onChange={ev => setUrlColumn (ev.target.value)}> <label>
URL 列
<select
value={urlColumn}
onChange={ev => setUrlColumn (ev.target.value)}>
<option value="">選択してください</option> <option value="">選択してください</option>
{columns.map ((column, index) => <option key={column.key} value={index}>{column.label}</option>)} {columns.map ((column, index) => (
</select></label> <option key={column.key} value={index}>
{['title', 'thumbnail_base', 'original_created_from', 'original_created_before', 'duration', 'tags', 'parent_post_ids'].map (field => {column.label}
<label key={field}>{field}<select value={mappings[field]?.columns?.[0] ?? ''} </option>))}
onChange={ev => setMappings (current => ({ ...current, [field]: { columns: ev.target.value === '' ? [] : [ev.target.value] } }))}> </select>
<option value="">指定なし</option> </label>
{columns.map ((column, index) => <option key={column.key} value={index}>{column.label}</option>)} {MAPPING_FIELDS.map (field =>
</select></label>)} renderMappingSelect (columns,
<div className="overflow-x-auto"><table className="text-sm"><thead><tr>{columns.map (column => <th key={column.key}>{column.label}</th>)}</tr></thead><tbody>{parsed.rows.slice (0, 5).map (row => <tr key={row.sourceRow}>{row.values.map ((value, index) => <td key={index}>{value}</td>)}</tr>)}</tbody></table></div> field,
<Button type="button" variant="outline" onClick={() => setPhase ('source')}>入力元へ戻る</Button> mappings[field]?.columns?.[0] ?? '',
<Button type="button" onClick={preview} disabled={loading || urlColumn === ''}>投稿情報を取得して確認</Button> updateMapping))}
</Form>) : phase === 'preview' ? ( <div className="overflow-x-auto">
<table className="text-sm">
<thead>
<tr>
{columns.map (column => (
<th key={column.key}>{column.label}</th>))}
</tr>
</thead>
<tbody>
{parsed.rows.slice (0, 5).map (row => (
<tr key={row.sourceRow}>
{row.values.map ((value, index) => (
<td key={index}>{value}</td>))}
</tr>))}
</tbody>
</table>
</div>
<Button
type="button"
variant="outline"
onClick={() => setPhase ('source')}>
入力元へ戻る
</Button>
<Button
type="button"
onClick={preview}
disabled={loading || urlColumn === ''}>
投稿情報を取得して確認
</Button>
</Form>)
: phase === 'preview'
? (
<div className="space-y-4"> <div className="space-y-4">
<div className="flex flex-wrap gap-3 rounded border p-3 dark:border-neutral-700"> <div className="flex flex-wrap gap-3 rounded border p-3 dark:border-neutral-700">
<label>URL 列 <select value={urlColumn} onChange={ev => setUrlColumn (ev.target.value)}> <label>
{columns.map ((column, index) => <option key={column.key} value={index}>{column.label}</option>)} URL 列
</select></label> <select
<label>既存投稿 <select value={existing} onChange={ev => updateExisting (ev.target.value as typeof existing)}> value={urlColumn}
<option value="skip">スキップ</option><option value="error">エラー</option> onChange={ev => setUrlColumn (ev.target.value)}>
</select></label> {columns.map ((column, index) => (
{['title', 'thumbnail_base', 'original_created_from', 'original_created_before', 'duration', 'tags', 'parent_post_ids'].map (field => <option key={column.key} value={index}>
<label key={field}>{field}<select value={mappings[field]?.columns?.[0] ?? ''} {column.label}
onChange={ev => setMappings (current => ({ ...current, [field]: { columns: ev.target.value === '' ? [] : [ev.target.value] } }))}> </option>))}
<option value="">指定なし</option>{columns.map ((column, index) => <option key={column.key} value={index}>{column.label}</option>)} </select>
</select></label>)} </label>
<Button type="button" variant="outline" onClick={preview} disabled={loading}>自動取得を更新</Button> <label>
既存投稿
<select
value={existing}
onChange={ev => updateExisting (ev.target.value as typeof existing)}>
<option value="skip">スキップ</option>
<option value="error">エラー</option>
</select>
</label>
{MAPPING_FIELDS.map (field =>
renderMappingSelect (columns,
field,
mappings[field]?.columns?.[0] ?? '',
updateMapping))}
<Button
type="button"
variant="outline"
onClick={preview}
disabled={loading}>
自動取得を更新
</Button>
</div> </div>
<div className="grid gap-3 md:grid-cols-2">{rows.map ((row, index) => <RowCard key={row.sourceRow} row={row} index={index} update={updateAttribute} updateURL={updateURL}/>)}</div> <div className="grid gap-3 md:grid-cols-2">
<Button type="button" variant="outline" onClick={() => setPhase ('mapping')}>列割当てへ戻る</Button> {rows.map ((row, index) => (
<Button type="button" onClick={submit} disabled={loading}>有効な投稿を一括登録</Button> <RowCard
</div>) : ( key={row.sourceRow}
<div className="space-y-3"><p>インポート処理が完了しました。</p> row={row}
{results.map (result => <div key={result.sourceRow} className="rounded border p-2 dark:border-neutral-700">行 {result.sourceRow}：{result.status} {result.post && <PrefetchLink to={`/posts/${result.post.id}`}>投稿を開く</PrefetchLink>}<FieldError messages={Object.values (result.errors ?? { }).flat ()}/></div>)} index={index}
<Button type="button" onClick={() => setPhase ('preview')}>結果を確認</Button> update={updateAttribute}
<Button type="button" variant="outline" onClick={() => setPhase ('preview')}>失敗行を修正</Button> updateURL={updateURL}/>))}
<Button type="button" variant="outline" onClick={() => setPhase ('source')}>新しい入力元を選ぶ</Button> </div>
<Button
type="button"
variant="outline"
onClick={() => setPhase ('mapping')}>
列割当てへ戻る
</Button>
<Button
type="button"
onClick={submit}
disabled={loading}>
有効な投稿を一括登録
</Button>
</div>)
: (
<div className="space-y-3">
<p>インポート処理が完了しました．</p>
{results.map (result => (
<div
key={result.sourceRow}
className="rounded border p-2 dark:border-neutral-700">
行 {result.sourceRow}：{result.status}{' '}
{result.post && (
<PrefetchLink to={`/posts/${ result.post.id }`}>
投稿を開く
</PrefetchLink>)}
<FieldError messages={Object.values (result.errors ?? { }).flat ()}/>
</div>))}
<Button
type="button"
onClick={() => setPhase ('preview')}>
結果を確認
</Button>
<Button
type="button"
variant="outline"
onClick={() => setPhase ('preview')}>
失敗行を修正
</Button>
<Button
type="button"
variant="outline"
onClick={() => setPhase ('source')}>
新しい入力元を選ぶ
</Button>
</div>)} </div>)}
</MainArea>) </MainArea>)
} }
const RowCard = ({ row, index, update, updateURL }: { row: ImportRow, index: number, update: (index: number, field: string, value: string) => void, updateURL: (index: number, value: string) => void }) => ( const RowCard = (
<article className="space-y-2 rounded border bg-white p-3 text-neutral-900 dark:border-neutral-700 dark:bg-neutral-900 dark:text-neutral-100"> { row, index, update, updateURL }: {
<p>行 {row.sourceRow}・{row.status}</p> row: ImportRow
{row.importStatus === 'created' ? <p className="rounded bg-green-100 p-2 dark:bg-green-900">登録済み。この行は編集・再登録できません。{row.createdPostId && <PrefetchLink to={`/posts/${row.createdPostId}`}>投稿を開く</PrefetchLink>}</p> : <> index: number
<EditableValue label="URL" value={row.url} origin={row.provenance.url} onCommit={value => updateURL (index, value)}/> update: (index: number, field: string, value: string) => void
<EditableValue label="タイトル" value={row.attributes.title ?? ''} origin={row.provenance.title} onCommit={value => update (index, 'title', value)}/> updateURL: (index: number, value: string) => void },
<EditableValue label="サムネール基底 URL" value={row.attributes.thumbnailBase ?? ''} origin={row.provenance.thumbnailBase} onCommit={value => update (index, 'thumbnailBase', value)}/> ) => (
<article
className="space-y-2 rounded border bg-white p-3 text-neutral-900
dark:border-neutral-700 dark:bg-neutral-900 dark:text-neutral-100">
<p>
行 {row.sourceRow}・{row.status}
</p>
{row.importStatus === 'created'
? (
<p className="rounded bg-green-100 p-2 dark:bg-green-900">
登録済み．この行は編輯・再登録できません．
{row.createdPostId && (
<PrefetchLink to={`/posts/${ row.createdPostId }`}>
投稿を開く
</PrefetchLink>)}
</p>)
: (
<>
<EditableValue
label="URL"
value={row.url}
origin={row.provenance.url}
onCommit={value => updateURL (index, value)}/>
<EditableValue
label="タイトル"
value={row.attributes.title ?? ''}
origin={row.provenance.title}
onCommit={value => update (index, 'title', value)}/>
<EditableValue
label="サムネール基底 URL"
value={row.attributes.thumbnailBase ?? ''}
origin={row.provenance.thumbnailBase}
onCommit={value => update (index, 'thumbnailBase', value)}/>
<ThumbnailPreview url={row.attributes.thumbnailBase ?? ''}/> <ThumbnailPreview url={row.attributes.thumbnailBase ?? ''}/>
<EditableValue label="作成日時 From" value={row.attributes.originalCreatedFrom ?? ''} origin={row.provenance.originalCreatedFrom} onCommit={value => update (index, 'originalCreatedFrom', value)}/> <EditableValue
<EditableValue label="作成日時 Before" value={row.attributes.originalCreatedBefore ?? ''} origin={row.provenance.originalCreatedBefore} onCommit={value => update (index, 'originalCreatedBefore', value)}/> label="作成日時 From"
<EditableValue label="動画時間" value={String (row.attributes.duration ?? '')} origin={row.provenance.duration} onCommit={value => update (index, 'duration', value)}/> value={row.attributes.originalCreatedFrom ?? ''}
<EditableValue label="タグ" value={row.attributes.tags ?? ''} origin={row.provenance.tags} onCommit={value => update (index, 'tags', value)}/> origin={row.provenance.originalCreatedFrom}
<EditableValue label="親投稿" value={row.attributes.parentPostIds ?? ''} origin={row.provenance.parentPostIds} onCommit={value => update (index, 'parentPostIds', value)}/> onCommit={value => update (index, 'originalCreatedFrom', value)}/>
</>} <EditableValue
<FieldError messages={[...row.warnings, label="作成日時 Before"
...Object.values (row.validationErrors ?? { }).flat (), value={row.attributes.originalCreatedBefore ?? ''}
...Object.values (row.importErrors ?? { }).flat ()]}/> origin={row.provenance.originalCreatedBefore}
onCommit={value => update (index, 'originalCreatedBefore', value)}/>
<EditableValue
label="動画時間"
value={String (row.attributes.duration ?? '')}
origin={row.provenance.duration}
onCommit={value => update (index, 'duration', value)}/>
<EditableValue
label="タグ"
value={row.attributes.tags ?? ''}
origin={row.provenance.tags}
onCommit={value => update (index, 'tags', value)}/>
<EditableValue
label="親投稿"
value={row.attributes.parentPostIds ?? ''}
origin={row.provenance.parentPostIds}
onCommit={value => update (index, 'parentPostIds', value)}/>
</>)}
<FieldError messages={rowMessages (row)}/>
</article>) </article>)
const EditableValue = ({ label, value, origin, onCommit }: { label: string, value: string, origin: string | undefined, onCommit: (value: string) => void }) => {
const EditableValue = (
{ label, value, origin, onCommit }: {
label: string
value: string
origin: Origin | undefined
onCommit: (value: string) => void },
) => {
const [editing, setEditing] = useState (false) const [editing, setEditing] = useState (false)
const [draft, setDraft] = useState (value) const [draft, setDraft] = useState (value)
const finished = useRef (false) const finished = useRef (false)
useEffect (() => { if (!editing) setDraft (value) }, [editing, value])
const originLabel = { automatic: '自動取得', mapped: '入力データ', fixed: '固定値', manual: '手修正' }[origin ?? 'automatic'] useEffect (() => {
const begin = () => { finished.current = false; setDraft (value); setEditing (true) } if (!(editing))
const commit = () => { if (finished.current) return; finished.current = true; setEditing (false); if (draft !== value) onCommit (draft) } setDraft (value)
const cancel = () => { if (finished.current) return; finished.current = true; setDraft (value); setEditing (false) } }, [editing, value])
return <div className={origin === 'manual' ? 'rounded bg-amber-100 p-1 dark:bg-amber-900' : 'p-1'}>
<span className="text-xs">{label}・{originLabel}</span> const begin = () => {
{editing ? <input autoFocus value={draft} onChange={ev => setDraft (ev.target.value)} onBlur={commit} finished.current = false
onKeyDown={ev => { if (ev.key === 'Escape') cancel (); if (ev.key === 'Enter') { ev.preventDefault (); commit () } } } className={inputClass (false)}/> setDraft (value)
: <button type="button" className="block w-full text-left" onDoubleClick={begin} onClick={begin}>{value || '（未指定）'}　編輯</button>} setEditing (true)
</div>
} }
const commit = () => {
if (finished.current)
return
finished.current = true
setEditing (false)
if (draft !== value)
onCommit (draft)
}
const cancel = () => {
if (finished.current)
return
finished.current = true
setDraft (value)
setEditing (false)
}
return (
<div
className={
origin === 'manual'
? 'rounded bg-amber-100 p-1 dark:bg-amber-900'
: 'p-1'
}>
<span className="text-xs">
{label}・{ORIGIN_LABELS[origin ?? 'automatic']}
</span>
{editing
? (
<input
autoFocus
value={draft}
onBlur={commit}
onChange={ev => setDraft (ev.target.value)}
onKeyDown={ev => {
if (ev.key === 'Escape')
cancel ()
if (ev.key === 'Enter')
{
ev.preventDefault ()
commit ()
}
}}
className={inputClass (false)}/>)
: (
<button
type="button"
className="block w-full text-left"
onClick={begin}
onDoubleClick={begin}>
{value || '（未指定）'}　編輯
</button>)}
</div>)
}
const ThumbnailPreview = ({ url }: { url: string }) => { const ThumbnailPreview = ({ url }: { url: string }) => {
const [failed, setFailed] = useState (false) const [failed, setFailed] = useState (false)
useEffect (() => setFailed (false), [url])
if (!url) useEffect (() => {
setFailed (false)
}, [url])
if (!(url))
return <p className="text-sm">サムネール未指定</p> return <p className="text-sm">サムネール未指定</p>
if (failed) if (failed)
return <p className="text-sm text-red-600 dark:text-red-300">サムネールを表示できません</p> {
return <img src={url} alt="サムネール" className="max-h-32" onError={() => setFailed (true)}/> return (
<p className="text-sm text-red-600 dark:text-red-300">
サムネールを表示できません
</p>)
}
return (
<img
src={url}
alt="サムネール"
className="max-h-32"
onError={() => setFailed (true)}/>)
} }
export default PostImportPage export default PostImportPage
frontend/src/pages/posts/PostNewPage.tsx
+23 -10
ファイルの表示
@@ -25,7 +25,13 @@ import type { User } from '@/types'
type Props = { user: User | null } type Props = { user: User | null }
type PostFormField = type PostFormField =
'url' | 'title' | 'tags' | 'parentPostIds' | 'videoMs' | 'originalCreatedAt' | 'thumbnail' 'url'
| 'title'
| 'tags'
| 'parentPostIds'
| 'videoMs'
| 'originalCreatedAt'
| 'thumbnail'
const PostNewPage: FC<Props> = ({ user }) => { const PostNewPage: FC<Props> = ({ user }) => {
@@ -36,8 +42,10 @@ const PostNewPage: FC<Props> = ({ user }) => {
const { baseErrors, fieldErrors, clearValidationErrors, applyValidationError } = const { baseErrors, fieldErrors, clearValidationErrors, applyValidationError } =
useValidationErrors<PostFormField> () useValidationErrors<PostFormField> ()
const [originalCreatedBefore, setOriginalCreatedBefore] = useState<string | null> (null) const [originalCreatedBefore, setOriginalCreatedBefore] =
const [originalCreatedFrom, setOriginalCreatedFrom] = useState<string | null> (null) useState<string | null> (null)
const [originalCreatedFrom, setOriginalCreatedFrom] =
useState<string | null> (null)
const [parentPostIds, setParentPostIds] = useState ('') const [parentPostIds, setParentPostIds] = useState ('')
const [tags, setTags] = useState ('') const [tags, setTags] = useState ('')
const [duration, setDuration] = useState ('') const [duration, setDuration] = useState ('')
@@ -50,7 +58,9 @@ const PostNewPage: FC<Props> = ({ user }) => {
const thumbnailPreviewRef = useRef ('') const thumbnailPreviewRef = useRef ('')
const videoFlg = const videoFlg =
useMemo (() => tags.split (/\s+/).some (tag => tag.replace (/\[.*\]$/, '') === '動画'), useMemo (() =>
tags.split (/\s+/).some (
tag => tag.replace (/\[.*\]$/, '') === '動画'),
[tags]) [tags])
const handleSubmit = async () => { const handleSubmit = async () => {
@@ -72,14 +82,15 @@ const PostNewPage: FC<Props> = ({ user }) => {
try try
{ {
await apiPost ('/posts', formData, { headers: { 'Content-Type': 'multipart/form-data' } }) await apiPost ('/posts', formData,
{ headers: { 'Content-Type': 'multipart/form-data' } })
toast ({ title: '投稿成功!' }) toast ({ title: '投稿成功!' })
navigate ('/posts') navigate ('/posts')
} }
catch (e) catch (e)
{ {
applyValidationError (e) applyValidationError (e)
toast ({ title: '投稿失敗', description: '入力を確認してください。' }) toast ({ title: '投稿失敗', description: '入力を確認してください．' })
} }
} }
@@ -87,7 +98,8 @@ const PostNewPage: FC<Props> = ({ user }) => {
setTitleLoading (true) setTitleLoading (true)
try try
{ {
const data = await apiGet<{ title: string }> ('/preview/title', { params: { url } }) const data = await apiGet<{ title: string }> ('/preview/title',
{ params: { url } })
setTitle (data.title || '') setTitle (data.title || '')
} }
finally finally
@@ -105,7 +117,8 @@ const PostNewPage: FC<Props> = ({ user }) => {
try try
{ {
const data = await apiGet<Blob> ('/preview/thumbnail', const data = await apiGet<Blob> ('/preview/thumbnail',
{ params: { url }, responseType: 'blob' }) { params: { url },
responseType: 'blob' })
const imageURL = URL.createObjectURL (data) const imageURL = URL.createObjectURL (data)
setThumbnailPreview (imageURL) setThumbnailPreview (imageURL)
setThumbnailFile (new File ([data], setThumbnailFile (new File ([data],
@@ -163,7 +176,7 @@ const PostNewPage: FC<Props> = ({ user }) => {
<Button <Button
type="button" type="button"
variant="outline" variant="outline"
onClick={() => void fetchTitle ()} onClick={fetchTitle}
disabled={!(url) || titleLoading}> disabled={!(url) || titleLoading}>
取得 取得
</Button> </Button>
@@ -180,7 +193,7 @@ const PostNewPage: FC<Props> = ({ user }) => {
<Button <Button
type="button" type="button"
variant="outline" variant="outline"
onClick={() => void fetchThumbnail ()} onClick={fetchThumbnail}
disabled={!(url) || thumbnailLoading}> disabled={!(url) || thumbnailLoading}>
取得 取得
</Button> </Button>