Reviewed-on: #413 Co-authored-by: miteruzo <miteruzo@naver.com> Co-committed-by: miteruzo <miteruzo@naver.com>
このコミットはPull リクエスト #413 でマージされました．
このコミットが含まれているのは:
+186
@@ -67,6 +67,192 @@ pass or the remaining failure is clearly blocked.
|
||||
Before changing behavior, inspect the matching route, controller, model,
|
||||
service, representation, and spec.
|
||||
|
||||
## Shared backend systems
|
||||
|
||||
Before adding backend behaviour, search the existing backend first. At minimum,
|
||||
check these locations:
|
||||
|
||||
- `app/controllers`
|
||||
- `app/controllers/concerns`
|
||||
- `app/models`
|
||||
- `app/models/concerns`
|
||||
- `app/representations`
|
||||
- `app/services`
|
||||
- `app/services/*`
|
||||
- `app/jobs`
|
||||
- `lib`
|
||||
- `lib/tasks`
|
||||
- `config/initializers`
|
||||
|
||||
Do not infer commonality from directory names alone. Read the actual
|
||||
responsibility and representative usage sites.
|
||||
|
||||
### Controller reuse
|
||||
|
||||
Before adding logic to a controller, inspect:
|
||||
|
||||
- `ApplicationController` authentication, authorization, BAN, and IP BAN
|
||||
- existing render and validation-error helpers
|
||||
- existing param parsing
|
||||
- controller concerns
|
||||
- the controller for the same resource
|
||||
- existing services
|
||||
- existing representations
|
||||
|
||||
Keep controllers focused on:
|
||||
|
||||
- authentication and authorization
|
||||
- parameter intake
|
||||
- service and model invocation
|
||||
- HTTP status selection
|
||||
- representation selection
|
||||
|
||||
Do not reimplement these per controller when an existing path already owns
|
||||
them:
|
||||
|
||||
- authentication and role checks
|
||||
- validation error JSON
|
||||
- URL normalisation
|
||||
- tag normalisation
|
||||
- thumbnail handling
|
||||
- version recording
|
||||
- complex transactions
|
||||
- external HTTP fetching
|
||||
- response representation assembly
|
||||
|
||||
### Authentication, authorization, and BAN
|
||||
|
||||
Treat these as the canonical backend entrypoints:
|
||||
|
||||
- `ApplicationController#authenticate_user`
|
||||
- `current_user`
|
||||
- `X-Transfer-Code`
|
||||
- `reject_banned_ip_address!`
|
||||
- `reject_banned_user!`
|
||||
- `gte_member?`
|
||||
- `admin?`
|
||||
|
||||
Do not create feature-local permission services, role comparisons, or header
|
||||
parsing when the existing authentication boundary already owns the behaviour.
|
||||
If the current boundary is insufficient, extend it minimally instead of adding
|
||||
another permission path.
|
||||
|
||||
### Representations
|
||||
|
||||
If an endpoint for the same resource already uses `app/representations`, do not
|
||||
assemble a separate JSON shape directly inside the controller without first
|
||||
checking the existing representation contract.
|
||||
|
||||
Inspect at least:
|
||||
|
||||
- `PostRepr`
|
||||
- `TagRepr`
|
||||
- `MaterialRepr`
|
||||
- `TheatreRepr`
|
||||
- `UserRepr`
|
||||
- `WikiPageRepr`
|
||||
- `DeerjikistRepr`
|
||||
|
||||
When a lightweight response is genuinely different in purpose, keep it
|
||||
deliberate and compatible with the surrounding contracts. Do not force every
|
||||
identifier list into a large representation, but do not fork the same resource
|
||||
shape casually either.
|
||||
|
||||
### Domain services
|
||||
|
||||
When work touches multiple models, transactions, external APIs, file handling,
|
||||
history creation, or multi-step workflow, search `app/services` first.
|
||||
|
||||
At minimum, search for existing services in these responsibility areas:
|
||||
|
||||
- version recorder and versioning
|
||||
- wiki commit
|
||||
- YouTube or Google Drive API client
|
||||
- material sync or ZIP export
|
||||
- similarity calculation
|
||||
- theatre selection or skip finalisation
|
||||
- metadata, thumbnail, or file processing
|
||||
- URL normaliser or sanitisation
|
||||
- import or export
|
||||
- preview safety or HTTP fetch
|
||||
|
||||
Do not create a same-responsibility service under another namespace or another
|
||||
name. If an existing service is close, extend that API minimally instead of
|
||||
wrapping it in a feature-local service.
|
||||
|
||||
### Versioning
|
||||
|
||||
When a feature writes history, snapshots, or restore roots, search the existing
|
||||
versioning path first. At minimum, inspect:
|
||||
|
||||
- `VersionRecorder`
|
||||
- `PostVersionRecorder`
|
||||
- `TagVersionRecorder`
|
||||
- `TagVersioning`
|
||||
- `MaterialVersionRecorder`
|
||||
- `NicoTagVersionRecorder`
|
||||
- `WikiVersionRecorder`
|
||||
|
||||
Do not implement history writes in controllers, callbacks, or ad hoc feature
|
||||
services when the recorder layer already owns the transaction boundary and
|
||||
meaning.
|
||||
|
||||
### Normalisation, sanitisation, and parsing
|
||||
|
||||
For URLs, tag names, times, video durations, identifiers, and paths, search the
|
||||
existing normaliser, sanitisation rule, parser, and model-callback path first.
|
||||
|
||||
Do not let frontend, controller, service, and model each invent different rules
|
||||
for the same value. Use one canonical normalisation path and keep input
|
||||
validation distinct from pre-persistence normalisation.
|
||||
|
||||
### External HTTP and URL safety
|
||||
|
||||
When fetching external URLs, reuse the existing preview-safety stack. Search at
|
||||
least for:
|
||||
|
||||
- URL safety
|
||||
- redirect validation
|
||||
- response size limits
|
||||
- timeouts
|
||||
- network failure mapping
|
||||
- HTML metadata extraction
|
||||
- known-site extraction
|
||||
- thumbnail fetching
|
||||
|
||||
Do not add direct `Net::HTTP`, `Faraday`, or equivalent feature-local HTTP code
|
||||
that reimplements SSRF checks, redirect restrictions, size limits, or timeouts.
|
||||
If the current fetcher is insufficient, extend its existing safety contract.
|
||||
|
||||
### Storage, files, and Active Storage
|
||||
|
||||
When handling files, thumbnails, ZIP output, object storage, or Active Storage
|
||||
blobs, inspect existing storage helpers, exporters, thumbnail generators, and
|
||||
checksum helpers first. Do not reimplement the same attach, export path,
|
||||
download, resize, or checksum flow in a controller or one-off service.
|
||||
|
||||
### Concerns
|
||||
|
||||
Do not create controller or model concerns merely because some code is shared.
|
||||
Use a concern only when multiple classes share the same lifecycle, macro,
|
||||
callback, or tightly cohesive behaviour. Utility collections belong in explicit
|
||||
objects or services, not in `CommonConcern`, `SharedMethods`, or `Utils`.
|
||||
|
||||
### Model boundaries
|
||||
|
||||
Model-specific invariants, associations, validations, and normalisation may
|
||||
live in the model. Multi-model workflow, external access, complex transaction
|
||||
flow, and feature orchestration belong in services. Do not hide feature
|
||||
workflow in model callbacks.
|
||||
|
||||
### Transactions, locking, and race handling
|
||||
|
||||
If transactions, locking, idempotency, or race recovery already exist in a
|
||||
service or model method, do not add a second implementation in a controller or
|
||||
new service. Inspect the existing transaction boundary first, avoid wrapping
|
||||
the same operation in needless nested transactions, and handle unique-constraint
|
||||
races according to the target constraint's business meaning.
|
||||
|
||||
## Ruby style
|
||||
|
||||
- Prefer precise, minimal changes.
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
class PostsController < ApplicationController
|
||||
Event = Struct.new(:post, :tag, :user, :change_type, :timestamp, keyword_init: true)
|
||||
MAX_BULK_REQUEST_BYTES = 40 * 1024 * 1024
|
||||
|
||||
class VideoMsParseError < ArgumentError
|
||||
;
|
||||
@@ -114,6 +115,77 @@ class PostsController < ApplicationController
|
||||
render json: PostRepr.base(post, current_user)
|
||||
end
|
||||
|
||||
def metadata
|
||||
return head :unauthorized unless current_user
|
||||
return head :forbidden unless current_user.gte_member?
|
||||
return render_bad_request('URL は必須です．') if params[:url].blank?
|
||||
|
||||
normal_url = PostUrlNormaliser.normalise(params[:url].to_s)
|
||||
return render_validation_error(fields: { url: ['URL が不正です．'] }) if normal_url.blank?
|
||||
|
||||
Preview::UrlSafety.validate(normal_url)
|
||||
existing_post = Post.with_attached_thumbnail.find_by(url: normal_url)
|
||||
if existing_post.present?
|
||||
return render json: {
|
||||
url: normal_url,
|
||||
title: nil,
|
||||
thumbnail_base: nil,
|
||||
tags: nil,
|
||||
display_tags: [],
|
||||
original_created_from: nil,
|
||||
original_created_before: nil,
|
||||
duration: nil,
|
||||
video_ms: nil,
|
||||
field_warnings: { },
|
||||
base_warnings: [],
|
||||
existing_post_id: existing_post.id,
|
||||
existing_post: compact_post(existing_post.id) }
|
||||
end
|
||||
|
||||
metadata = PostMetadataFetcher.fetch(normal_url)
|
||||
field_warnings = { }
|
||||
field_warnings[:title] = ['タイトルを取得できませんでした．'] if metadata[:title].blank?
|
||||
if metadata[:thumbnail_base].blank?
|
||||
field_warnings[:thumbnail_base] = ['サムネールを取得できませんでした．']
|
||||
end
|
||||
|
||||
render json: {
|
||||
url: normal_url,
|
||||
title: metadata[:title],
|
||||
thumbnail_base: metadata[:thumbnail_base],
|
||||
tags: metadata[:tags],
|
||||
display_tags: metadata[:display_tags],
|
||||
original_created_from: metadata[:original_created_from],
|
||||
original_created_before: metadata[:original_created_before],
|
||||
duration: metadata[:duration],
|
||||
video_ms: metadata[:video_ms],
|
||||
field_warnings: field_warnings,
|
||||
base_warnings: [],
|
||||
existing_post_id: nil,
|
||||
existing_post: nil }
|
||||
rescue ArgumentError => e
|
||||
render_bad_request e.message
|
||||
rescue Preview::UrlSafety::UnsafeUrl => e
|
||||
render_validation_error fields: { url: [e.message] }
|
||||
rescue Preview::HttpFetcher::FetchFailed,
|
||||
Preview::HttpFetcher::FetchTimeout,
|
||||
Preview::HttpFetcher::ResponseTooLarge
|
||||
render json: {
|
||||
url: normal_url,
|
||||
title: nil,
|
||||
thumbnail_base: nil,
|
||||
tags: nil,
|
||||
display_tags: [],
|
||||
original_created_from: nil,
|
||||
original_created_before: nil,
|
||||
duration: nil,
|
||||
video_ms: nil,
|
||||
field_warnings: { url: ['自動取得に失敗しました．'] },
|
||||
base_warnings: [],
|
||||
existing_post_id: nil,
|
||||
existing_post: nil }
|
||||
end
|
||||
|
||||
def show
|
||||
post =
|
||||
Post
|
||||
@@ -142,48 +214,51 @@ class PostsController < ApplicationController
|
||||
return head :unauthorized unless current_user
|
||||
return head :forbidden unless current_user.gte_member?
|
||||
|
||||
# TODO: サイトに応じて thumbnail_base 設定
|
||||
title = params[:title].presence
|
||||
url = params[:url]
|
||||
thumbnail = params[:thumbnail]
|
||||
tag_names = params[:tags].to_s.split
|
||||
original_created_from = params[:original_created_from]
|
||||
original_created_before = params[:original_created_before]
|
||||
parent_post_ids = parse_parent_post_ids
|
||||
resized_thumbnail = thumbnail.present? ? Post.resized_thumbnail_attachment(thumbnail) : nil
|
||||
|
||||
post = Post.new(title:, url:, thumbnail_base: nil, uploaded_user: current_user,
|
||||
original_created_from:, original_created_before:)
|
||||
post.thumbnail.attach(resized_thumbnail) if resized_thumbnail
|
||||
|
||||
ApplicationRecord.transaction do
|
||||
post.save!
|
||||
|
||||
Tag.normalise_tags!(tag_names, deny_deprecated: true, with_sections: true) =>
|
||||
{ tags:, sections: }
|
||||
TagVersioning.record_tag_snapshots!(tags, created_by_user: current_user)
|
||||
|
||||
tags = Tag.expand_parent_tags(tags).reject(&:deprecated?)
|
||||
post.video_ms = normalise_video_ms(tags)
|
||||
validate_video_sections!(post.video_ms, sections)
|
||||
post.save!
|
||||
sync_post_tags!(post, tags, sections)
|
||||
|
||||
sync_parent_posts!(post, parent_post_ids)
|
||||
|
||||
PostVersionRecorder.record!(post:, event_type: :create, created_by_user: current_user)
|
||||
preflight = PostCreatePreflight.new(
|
||||
attributes: post_create_attributes,
|
||||
thumbnail: params[:thumbnail],
|
||||
host: request.base_url).run
|
||||
return render json: dry_run_json(preflight) if bool?(:dry)
|
||||
if preflight[:existing_post_id].present?
|
||||
post = Post.new(url: preflight[:url])
|
||||
post.errors.add :url, :taken
|
||||
return render_post_form_record_invalid post
|
||||
end
|
||||
|
||||
post = PostCreator.new(actor: current_user,
|
||||
attributes: post_create_attributes.merge(
|
||||
preflight.slice(
|
||||
:url,
|
||||
:title,
|
||||
:thumbnail_base,
|
||||
:tags,
|
||||
:parent_post_ids,
|
||||
:original_created_from,
|
||||
:original_created_before,
|
||||
:duration,
|
||||
:video_ms,
|
||||
:direct_tag_specs,
|
||||
:default_tag_specs,
|
||||
:snapshot_tag_specs,
|
||||
:post_tag_specs,
|
||||
:tag_sections,
|
||||
:normalised_parent_post_ids).symbolize_keys).merge(
|
||||
thumbnail: params[:thumbnail])).create!
|
||||
|
||||
post.reload
|
||||
render json: PostRepr.base(post), status: :created
|
||||
rescue PostCreatePreflight::ValidationFailed => e
|
||||
render_validation_error fields: e.fields, base: e.base_errors
|
||||
rescue Tag::NicoTagNormalisationError
|
||||
render_validation_error fields: { tags: 'ニコニコ・タグは直接指定できません．' }
|
||||
rescue Tag::DeprecatedTagNormalisationError
|
||||
render_unprocessable_entity '廃止済みタグは付与できません．', field: :tags
|
||||
rescue Tag::SectionLiteralParseError
|
||||
render_validation_error fields: { tags: ['タグ区間の記法が不正です．'] }
|
||||
rescue VideoMsParseError
|
||||
rescue PostCreator::VideoMsParseError
|
||||
render_validation_error fields: { video_ms: ['動画時間の記法が不正です．'] }
|
||||
rescue Post::RemoteThumbnailFetchFailed
|
||||
render_validation_error fields: { thumbnail_base: ['サムネイル画像の取得に失敗しました．'] }
|
||||
rescue MiniMagick::Error
|
||||
render_validation_error fields: { thumbnail: ['サムネイル画像の変換に失敗しました．'] }
|
||||
rescue ArgumentError => e
|
||||
@@ -192,6 +267,25 @@ class PostsController < ApplicationController
|
||||
render_post_form_record_invalid e.record
|
||||
end
|
||||
|
||||
def bulk
|
||||
return head :unauthorized unless current_user
|
||||
return head :forbidden unless current_user.gte_member?
|
||||
return head :unsupported_media_type unless request.content_mime_type == Mime[:multipart_form]
|
||||
return head :payload_too_large if request.content_length.to_i > MAX_BULK_REQUEST_BYTES
|
||||
posts = parse_bulk_posts_manifest
|
||||
thumbnails = parse_bulk_thumbnails(posts.length)
|
||||
result = PostBulkCreator.new(
|
||||
actor: current_user,
|
||||
posts:,
|
||||
thumbnails:,
|
||||
host: request.base_url).run
|
||||
render json: result
|
||||
rescue JSON::ParserError
|
||||
render_bad_request 'posts manifest の JSON が不正です．'
|
||||
rescue ArgumentError => e
|
||||
render_validation_error base: [e.message]
|
||||
end
|
||||
|
||||
def viewed
|
||||
return head :unauthorized unless current_user
|
||||
|
||||
@@ -486,6 +580,79 @@ class PostsController < ApplicationController
|
||||
}.uniq
|
||||
end
|
||||
|
||||
def post_create_attributes
|
||||
{ title: params[:title],
|
||||
url: params[:url],
|
||||
thumbnail_base: params[:thumbnail_base],
|
||||
tags: params[:tags],
|
||||
original_created_from: params[:original_created_from],
|
||||
original_created_before: params[:original_created_before],
|
||||
parent_post_ids: parse_parent_post_ids,
|
||||
video_ms: params[:video_ms],
|
||||
duration: params[:duration] }
|
||||
end
|
||||
|
||||
def parse_bulk_posts_manifest
|
||||
manifest = params[:posts]
|
||||
raise ArgumentError, 'posts は必須です．' if manifest.blank?
|
||||
raise ArgumentError, 'posts は JSON 文字列で指定してください．' unless manifest.is_a?(String)
|
||||
|
||||
posts = JSON.parse(manifest)
|
||||
raise ArgumentError, 'posts は配列で指定してください．' unless posts.is_a?(Array)
|
||||
raise ArgumentError, '投稿件数は 1 件以上必要です．' if posts.empty?
|
||||
raise ArgumentError, '投稿件数が多すぎます．' if posts.length > 100
|
||||
raise ArgumentError, 'posts 要素の形式が不正です．' unless posts.all? { _1.is_a?(Hash) }
|
||||
|
||||
posts
|
||||
end
|
||||
|
||||
def parse_bulk_thumbnails post_count
|
||||
thumbnails = { }
|
||||
raw = params[:thumbnails]
|
||||
return thumbnails if raw.blank?
|
||||
raise ArgumentError, 'thumbnail key が不正です．' unless raw.respond_to?(:to_unsafe_h)
|
||||
|
||||
raw.to_unsafe_h.each do |key, value|
|
||||
raise ArgumentError, 'thumbnail key が不正です．' unless key.to_s.match?(/\A\d+\z/)
|
||||
|
||||
index = Integer(key, 10)
|
||||
raise ArgumentError, 'thumbnail index が範囲外です．' if index.negative? || index >= post_count
|
||||
raise ArgumentError, 'thumbnail index が重複しています．' if thumbnails.key?(index)
|
||||
unless value.is_a?(ActionDispatch::Http::UploadedFile)
|
||||
raise ArgumentError, 'thumbnail upload が不正です．'
|
||||
end
|
||||
|
||||
thumbnails[index] = value
|
||||
end
|
||||
|
||||
thumbnails
|
||||
end
|
||||
|
||||
def compact_post post_id
|
||||
return nil if post_id.blank?
|
||||
|
||||
post = Post.with_attached_thumbnail.find_by(id: post_id)
|
||||
PostCompactRepr.base(post, host: request.base_url)
|
||||
end
|
||||
|
||||
def dry_run_json preflight
|
||||
preflight.slice(
|
||||
:url,
|
||||
:title,
|
||||
:thumbnail_base,
|
||||
:tags,
|
||||
:display_tags,
|
||||
:parent_post_ids,
|
||||
:original_created_from,
|
||||
:original_created_before,
|
||||
:duration,
|
||||
:video_ms,
|
||||
:field_warnings,
|
||||
:base_warnings,
|
||||
:existing_post_id,
|
||||
:existing_post)
|
||||
end
|
||||
|
||||
def sync_parent_posts! post, parent_post_ids
|
||||
if parent_post_ids.include?(post.id)
|
||||
post.errors.add :parent_post_ids, '自分自身を親投稿にはできません．'
|
||||
@@ -565,6 +732,7 @@ class PostsController < ApplicationController
|
||||
|
||||
def post_incoming_snapshot title:, original_created_from:, original_created_before:,
|
||||
tag_names:, video_ms_param:, duration_param:, parent_post_ids:
|
||||
validate_original_created_values!(original_created_from, original_created_before)
|
||||
Tag.normalise_tags!(tag_names, with_tagme: false, deny_deprecated: true,
|
||||
with_sections: true) =>
|
||||
{ tags:, sections: }
|
||||
@@ -602,6 +770,23 @@ class PostsController < ApplicationController
|
||||
value.to_s
|
||||
end
|
||||
|
||||
def validate_original_created_values! original_created_from, original_created_before
|
||||
candidate = Post.new(
|
||||
url: 'https://example.invalid/original-created-validation',
|
||||
original_created_from:,
|
||||
original_created_before:)
|
||||
candidate.valid?
|
||||
fields = [:original_created_from, :original_created_before, :original_created_at]
|
||||
relevant_errors = candidate.errors.select { fields.include?(_1.attribute) }
|
||||
return if relevant_errors.empty?
|
||||
|
||||
invalid_post = Post.new
|
||||
relevant_errors.each { |error|
|
||||
invalid_post.errors.add(error.attribute, error.message)
|
||||
}
|
||||
raise ActiveRecord::RecordInvalid, invalid_post
|
||||
end
|
||||
|
||||
def section_literal section
|
||||
"[#{ Post.ms_to_time(section[0]) }-#{ section[1] ? Post.ms_to_time(section[1]) : '' }]"
|
||||
end
|
||||
|
||||
@@ -18,14 +18,12 @@ class PreviewController < ApplicationController
|
||||
def thumbnail
|
||||
return render_bad_request('URL は必須です．') if params[:url].blank?
|
||||
|
||||
image = MiniMagick::Image.read(Preview::ThumbnailFetcher.fetch(params[:url]))
|
||||
image.auto_orient
|
||||
image.resize '180x180>'
|
||||
image.format 'png'
|
||||
width, height = image.dimensions
|
||||
raise Preview::ThumbnailFetcher::GenerationFailed, 'サムネール画像の変換に失敗しました．' if width > 180 || height > 180
|
||||
|
||||
send_data image.to_blob, type: 'image/png', disposition: 'inline'
|
||||
attachment =
|
||||
Post.resized_thumbnail_attachment(
|
||||
StringIO.new(Preview::ThumbnailFetcher.fetch(params[:url])))
|
||||
send_data attachment[:io].read,
|
||||
type: attachment[:content_type],
|
||||
disposition: 'inline'
|
||||
rescue Preview::UrlSafety::UnsafeUrl => e
|
||||
render_bad_request(e.message)
|
||||
rescue Preview::HttpFetcher::FetchTimeout => e
|
||||
|
||||
+366
-23
@@ -1,20 +1,57 @@
|
||||
class Post < ApplicationRecord
|
||||
require 'date'
|
||||
require 'mini_magick'
|
||||
require 'nokogiri'
|
||||
require 'stringio'
|
||||
require 'timeout'
|
||||
|
||||
def self.resized_thumbnail_attachment(upload)
|
||||
class RemoteThumbnailFetchFailed < StandardError; end
|
||||
|
||||
ORIGINAL_CREATED_INVALID_MESSAGE = 'オリジナルの作成日時の形式が不正です．'.freeze
|
||||
ORIGINAL_CREATED_MINUTE_PRECISION_MESSAGE =
|
||||
'オリジナルの作成日時は分単位で入力してください．'.freeze
|
||||
ORIGINAL_CREATED_ORDER_MESSAGE = 'オリジナルの作成日時の順番がをかしぃです．'.freeze
|
||||
ORIGINAL_CREATED_MINIMUM_RANGE_MESSAGE =
|
||||
'オリジナルの作成日時の範囲は1分以上必要です．'.freeze
|
||||
REMOTE_SVG_CONTENT_TYPE = 'image/svg+xml'.freeze
|
||||
MAX_SVG_DIMENSION = 4_096
|
||||
MAX_SVG_PIXELS = 16_777_216
|
||||
THUMBNAIL_PROCESS_TIMEOUT = 5.seconds
|
||||
def self.resized_thumbnail_attachment(upload, content_type: nil)
|
||||
upload.rewind
|
||||
image = MiniMagick::Image.read(upload.read)
|
||||
image.resize '180x180'
|
||||
image.format 'jpg'
|
||||
bytes = upload.read
|
||||
blob = Timeout.timeout(THUMBNAIL_PROCESS_TIMEOUT) do
|
||||
image = image_for_thumbnail_upload(bytes, content_type:)
|
||||
image.auto_orient
|
||||
image.resize '180x180'
|
||||
image.format 'jpg'
|
||||
image.to_blob
|
||||
end
|
||||
|
||||
{ io: StringIO.new(image.to_blob),
|
||||
{ io: StringIO.new(blob),
|
||||
filename: 'resized_thumbnail.jpg',
|
||||
content_type: 'image/jpeg' }
|
||||
rescue Timeout::Error
|
||||
raise MiniMagick::Error, 'サムネイル画像の変換に失敗しました．'
|
||||
ensure
|
||||
upload.rewind
|
||||
end
|
||||
|
||||
def self.remote_thumbnail_attachment(raw_url)
|
||||
response = Preview::ThumbnailFetcher.fetch_image_response(raw_url)
|
||||
resized_thumbnail_attachment(
|
||||
StringIO.new(response.body),
|
||||
content_type: response.content_type)
|
||||
rescue Preview::UrlSafety::UnsafeUrl,
|
||||
Preview::ThumbnailFetcher::GenerationFailed,
|
||||
Preview::HttpFetcher::FetchFailed,
|
||||
Preview::HttpFetcher::FetchTimeout,
|
||||
Preview::HttpFetcher::ResponseTooLarge,
|
||||
Timeout::Error,
|
||||
MiniMagick::Error => e
|
||||
raise RemoteThumbnailFetchFailed, e.message
|
||||
end
|
||||
|
||||
belongs_to :uploaded_user, class_name: 'User', optional: true
|
||||
|
||||
has_many :post_tags, dependent: :destroy, inverse_of: :post
|
||||
@@ -101,7 +138,10 @@ class Post < ApplicationRecord
|
||||
end
|
||||
|
||||
def self.section_literal section
|
||||
"[#{ Post.ms_to_time(section.begin_ms) }-#{ section.end_ms ? Post.ms_to_time(section.end_ms) : '' }]"
|
||||
end_ms =
|
||||
section.end_ms ? Post.ms_to_time(section.end_ms) : ''
|
||||
|
||||
"[#{ Post.ms_to_time(section.begin_ms) }-#{ end_ms }]"
|
||||
end
|
||||
|
||||
def self.ms_to_time ms
|
||||
@@ -118,7 +158,11 @@ class Post < ApplicationRecord
|
||||
'%d:%02d' % [min, s]
|
||||
end
|
||||
|
||||
remainder_ms.positive? ? "#{ base }.#{ remainder_ms.to_s.rjust(3, '0') }" : base
|
||||
if remainder_ms.positive?
|
||||
"#{ base }.#{ remainder_ms.to_s.rjust(3, '0') }"
|
||||
else
|
||||
base
|
||||
end
|
||||
end
|
||||
|
||||
def snapshot_parent_post_ids = parents.order(:id).pluck(:id)
|
||||
@@ -140,19 +184,26 @@ class Post < ApplicationRecord
|
||||
thumbnail.attach(self.class.resized_thumbnail_attachment(StringIO.new(thumbnail.download)))
|
||||
end
|
||||
|
||||
def attach_thumbnail_from_url! raw_url
|
||||
thumbnail.attach(self.class.remote_thumbnail_attachment(raw_url))
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def validate_original_created_range
|
||||
f = original_created_from
|
||||
b = original_created_before
|
||||
return if f.blank? || b.blank?
|
||||
return if skip_original_created_validation?
|
||||
|
||||
f = Time.zone.parse(f) if String === f
|
||||
b = Time.zone.parse(b) if String === b
|
||||
return if !(f) || !(b)
|
||||
f = parse_original_created_value(:original_created_from)
|
||||
b = parse_original_created_value(:original_created_before)
|
||||
return if f.nil? || b.nil?
|
||||
|
||||
if f >= b
|
||||
errors.add :original_created_at, 'オリジナルの作成日時の順番がをかしぃです．'
|
||||
if b <= f
|
||||
errors.add :original_created_at, ORIGINAL_CREATED_ORDER_MESSAGE
|
||||
return
|
||||
end
|
||||
|
||||
if b - f < 1.minute
|
||||
errors.add :original_created_at, ORIGINAL_CREATED_MINIMUM_RANGE_MESSAGE
|
||||
end
|
||||
end
|
||||
|
||||
@@ -173,15 +224,307 @@ class Post < ApplicationRecord
|
||||
def normalise_url
|
||||
return if url.blank?
|
||||
|
||||
self.url = url.strip
|
||||
self.url = PostUrlNormaliser.normalise(url) || url.strip
|
||||
end
|
||||
|
||||
u = URI.parse(url)
|
||||
return unless u in URI::HTTP
|
||||
def self.image_for_thumbnail_upload(bytes, content_type: nil)
|
||||
if svg_content_type?(content_type) || svg_document_bytes?(bytes)
|
||||
return decode_svg_thumbnail(bytes)
|
||||
end
|
||||
|
||||
u.host = u.host.downcase if u.host
|
||||
u.path = u.path.sub(/\/\Z/, '') if u.path.present?
|
||||
self.url = PostUrlSanitisationRule.sanitise(u.to_s)
|
||||
rescue URI::InvalidURIError
|
||||
;
|
||||
raise MiniMagick::Error, 'サムネイル画像の形式が不正です．' unless raster_thumbnail_bytes?(bytes)
|
||||
|
||||
decode_raster_thumbnail(bytes)
|
||||
end
|
||||
|
||||
def self.raster_thumbnail_bytes?(bytes)
|
||||
raster_thumbnail_format(bytes).present?
|
||||
end
|
||||
|
||||
def self.remote_thumbnail_image_bytes?(bytes, content_type: nil)
|
||||
return true if svg_content_type?(content_type) || svg_document_bytes?(bytes)
|
||||
|
||||
raster_thumbnail_bytes?(bytes)
|
||||
end
|
||||
|
||||
def self.svg_content_type?(content_type)
|
||||
content_type.to_s.split(';', 2).first.to_s.downcase.strip == REMOTE_SVG_CONTENT_TYPE
|
||||
end
|
||||
|
||||
def self.svg_document_bytes?(bytes)
|
||||
document = Nokogiri::XML(
|
||||
bytes,
|
||||
nil,
|
||||
nil,
|
||||
Nokogiri::XML::ParseOptions::STRICT |
|
||||
Nokogiri::XML::ParseOptions::NONET)
|
||||
document.root&.name == 'svg'
|
||||
rescue Nokogiri::XML::SyntaxError
|
||||
false
|
||||
end
|
||||
|
||||
def self.decode_raster_thumbnail(bytes)
|
||||
MiniMagick::Image.read(bytes)
|
||||
end
|
||||
|
||||
def self.decode_svg_thumbnail(bytes)
|
||||
MiniMagick::Image.read(sanitised_svg_bytes(bytes))
|
||||
end
|
||||
|
||||
def self.raster_thumbnail_format(bytes)
|
||||
binary = bytes.to_s.b
|
||||
return 'jpeg' if binary.start_with?("\xFF\xD8\xFF".b)
|
||||
return 'png' if binary.start_with?("\x89PNG\r\n\x1A\n".b)
|
||||
return 'gif' if binary.start_with?('GIF87a'.b) || binary.start_with?('GIF89a'.b)
|
||||
return 'webp' if binary.bytesize >= 12 &&
|
||||
binary.start_with?('RIFF'.b) &&
|
||||
binary.byteslice(8, 4) == 'WEBP'
|
||||
|
||||
nil
|
||||
end
|
||||
|
||||
def self.sanitised_svg_bytes(bytes)
|
||||
parse_options =
|
||||
Nokogiri::XML::ParseOptions::STRICT |
|
||||
Nokogiri::XML::ParseOptions::NONET
|
||||
document = Nokogiri::XML(
|
||||
bytes,
|
||||
nil,
|
||||
nil,
|
||||
parse_options)
|
||||
root = document.root
|
||||
raise MiniMagick::Error, 'SVG が不正です．' if root == nil || root.name != 'svg'
|
||||
raise MiniMagick::Error, 'SVG が不正です．' if document.internal_subset != nil
|
||||
raise MiniMagick::Error, 'SVG が不正です．' if svg_uses_disallowed_features?(document)
|
||||
|
||||
width, height = svg_dimensions(root)
|
||||
raise MiniMagick::Error, 'SVG が大きすぎます．' if width == nil || height == nil
|
||||
if width > MAX_SVG_DIMENSION || height > MAX_SVG_DIMENSION || width * height > MAX_SVG_PIXELS
|
||||
raise MiniMagick::Error, 'SVG が大きすぎます．'
|
||||
end
|
||||
|
||||
document.to_xml
|
||||
rescue Nokogiri::XML::SyntaxError
|
||||
raise MiniMagick::Error, 'SVG が不正です．'
|
||||
end
|
||||
|
||||
def self.svg_uses_disallowed_features?(document)
|
||||
document.traverse.any? do |node|
|
||||
next false unless node.element?
|
||||
|
||||
name = node.name.to_s.downcase
|
||||
next true if name == 'script' || name == 'foreignobject'
|
||||
next style_contains_disallowed_urls?(node.text.to_s) if name == 'style'
|
||||
|
||||
node.attribute_nodes.any? do |attribute|
|
||||
attribute_name = attribute.name.to_s.downcase
|
||||
attribute_value = attribute.value.to_s
|
||||
attribute_name.start_with?('on') ||
|
||||
external_svg_reference?(attribute_name, attribute_value)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def self.external_svg_reference?(attribute_name, attribute_value)
|
||||
if ['href', 'xlink:href', 'src'].include?(attribute_name)
|
||||
return external_svg_url?(attribute_value)
|
||||
end
|
||||
return style_contains_disallowed_urls?(attribute_value) if attribute_name == 'style'
|
||||
return svg_url_function_disallowed?(attribute_value) if attribute_value.match?(/url\s*\(/i)
|
||||
|
||||
false
|
||||
end
|
||||
|
||||
def self.external_svg_url?(value)
|
||||
stripped = value.to_s.strip
|
||||
return false if stripped.blank? || stripped.start_with?('#')
|
||||
|
||||
true
|
||||
end
|
||||
|
||||
def self.style_contains_disallowed_urls?(value)
|
||||
text = value.to_s
|
||||
text.match?(/@import/i) || svg_url_function_disallowed?(text)
|
||||
end
|
||||
|
||||
def self.svg_url_function_disallowed?(value)
|
||||
value.to_s.scan(/url\s*\(([^)]*)\)/i).flatten.any? do |entry|
|
||||
reference =
|
||||
entry.to_s.strip
|
||||
.delete_prefix("'")
|
||||
.delete_prefix('"')
|
||||
.delete_suffix("'")
|
||||
.delete_suffix('"')
|
||||
reference.present? && !(reference.start_with?('#'))
|
||||
end
|
||||
end
|
||||
|
||||
def self.svg_dimensions(root)
|
||||
width = svg_length_to_pixels(root['width'])
|
||||
height = svg_length_to_pixels(root['height'])
|
||||
return [width, height] if width && height
|
||||
|
||||
view_box = root['viewBox'].to_s.strip.split(/\s+/).map { Float(_1) rescue nil }
|
||||
return [nil, nil] if view_box.length != 4 || view_box.any?(&:nil?)
|
||||
return [nil, nil] unless view_box[2].finite? && view_box[2].positive?
|
||||
return [nil, nil] unless view_box[3].finite? && view_box[3].positive?
|
||||
|
||||
[view_box[2], view_box[3]]
|
||||
end
|
||||
|
||||
def self.svg_length_to_pixels(value)
|
||||
return nil if value.blank?
|
||||
|
||||
matched = /\A([0-9]+(?:\.[0-9]+)?)(px)?\z/i.match(value.to_s.strip)
|
||||
return nil if matched == nil
|
||||
|
||||
pixels = Float(matched[1])
|
||||
return nil unless pixels.finite? && pixels.positive?
|
||||
|
||||
pixels
|
||||
rescue ArgumentError
|
||||
nil
|
||||
end
|
||||
|
||||
private_class_method :image_for_thumbnail_upload,
|
||||
:svg_content_type?,
|
||||
:decode_raster_thumbnail,
|
||||
:decode_svg_thumbnail,
|
||||
:raster_thumbnail_format,
|
||||
:sanitised_svg_bytes,
|
||||
:svg_uses_disallowed_features?,
|
||||
:external_svg_reference?,
|
||||
:external_svg_url?,
|
||||
:style_contains_disallowed_urls?,
|
||||
:svg_url_function_disallowed?,
|
||||
:svg_dimensions,
|
||||
:svg_length_to_pixels
|
||||
|
||||
def parse_original_created_value field
|
||||
raw_value = public_send("#{ field }_before_type_cast")
|
||||
value = public_send(field)
|
||||
return nil if raw_value.blank? && value.blank?
|
||||
|
||||
time =
|
||||
case raw_value
|
||||
when String
|
||||
parse_original_created_string(raw_value)
|
||||
when Time, ActiveSupport::TimeWithZone
|
||||
raw_value.in_time_zone
|
||||
else
|
||||
value&.in_time_zone
|
||||
end
|
||||
if time.nil?
|
||||
errors.add field, ORIGINAL_CREATED_INVALID_MESSAGE
|
||||
return nil
|
||||
end
|
||||
unless minute_precision_time?(time)
|
||||
errors.add field, ORIGINAL_CREATED_MINUTE_PRECISION_MESSAGE
|
||||
return nil
|
||||
end
|
||||
time
|
||||
end
|
||||
|
||||
def parse_original_created_string raw_value
|
||||
value = raw_value.to_s.strip
|
||||
return nil if value.blank?
|
||||
|
||||
match = value.match(/\A(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2})\z/)
|
||||
if match
|
||||
year = match[1].to_i
|
||||
month = match[2].to_i
|
||||
day = match[3].to_i
|
||||
hour = match[4].to_i
|
||||
minute = match[5].to_i
|
||||
return nil unless valid_original_created_components?(year, month, day, hour, minute, 0)
|
||||
|
||||
return Time.zone.local(year, month, day, hour, minute)
|
||||
end
|
||||
|
||||
match =
|
||||
value.match(
|
||||
/
|
||||
\A(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2})
|
||||
(?::(\d{2})(?:\.(\d+))?)?
|
||||
(Z|[+-]\d{2}:?\d{2})?
|
||||
\z
|
||||
/x)
|
||||
return nil if match.nil?
|
||||
|
||||
year = match[1].to_i
|
||||
month = match[2].to_i
|
||||
day = match[3].to_i
|
||||
hour = match[4].to_i
|
||||
minute = match[5].to_i
|
||||
second = match[6]&.to_i || 0
|
||||
fraction = match[7]
|
||||
offset = match[8]
|
||||
return nil unless valid_original_created_components?(year, month, day, hour, minute, second)
|
||||
return nil if offset.present? && !(valid_original_created_offset?(offset))
|
||||
|
||||
if offset.present?
|
||||
return Time.new(
|
||||
year,
|
||||
month,
|
||||
day,
|
||||
hour,
|
||||
minute,
|
||||
second + Rational(parse_original_created_nanoseconds(fraction), 1_000_000_000),
|
||||
normalise_original_created_offset(offset)).in_time_zone
|
||||
end
|
||||
|
||||
Time.zone.local(
|
||||
year,
|
||||
month,
|
||||
day,
|
||||
hour,
|
||||
minute,
|
||||
second).change(nsec: parse_original_created_nanoseconds(fraction))
|
||||
rescue ArgumentError, TypeError
|
||||
nil
|
||||
end
|
||||
|
||||
def minute_precision_time? value
|
||||
value.sec.zero? && value.nsec.zero?
|
||||
end
|
||||
|
||||
def valid_original_created_components? year, month, day, hour, minute, second
|
||||
return false unless Date.valid_date?(year, month, day)
|
||||
return false unless hour.between?(0, 23)
|
||||
return false unless minute.between?(0, 59)
|
||||
return false unless second.between?(0, 59)
|
||||
|
||||
true
|
||||
end
|
||||
|
||||
def valid_original_created_offset? value
|
||||
match = value.match(/\A([+-])(\d{2}):?(\d{2})\z/)
|
||||
return true if value == 'Z'
|
||||
return false if match.nil?
|
||||
|
||||
hours = match[2].to_i
|
||||
minutes = match[3].to_i
|
||||
hours.between?(0, 23) && minutes.between?(0, 59)
|
||||
end
|
||||
|
||||
def parse_original_created_nanoseconds value
|
||||
return 0 if value.blank?
|
||||
|
||||
digits = value[0, 9].ljust(9, '0')
|
||||
Integer(digits, 10)
|
||||
end
|
||||
|
||||
def normalise_original_created_offset value
|
||||
return '+00:00' if value == 'Z'
|
||||
|
||||
value.match?(/\A[+-]\d{2}:\d{2}\z/) ? value : "#{ value[0, 3] }:#{ value[3, 2] }"
|
||||
end
|
||||
|
||||
def skip_original_created_validation?
|
||||
return false if new_record?
|
||||
return false if will_save_change_to_original_created_from?
|
||||
return false if will_save_change_to_original_created_before?
|
||||
|
||||
true
|
||||
end
|
||||
end
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
|
||||
module PostCompactRepr
|
||||
module_function
|
||||
|
||||
def base post, host: nil
|
||||
return nil if post.nil?
|
||||
|
||||
PostRepr
|
||||
.common(post, host:)
|
||||
.slice(
|
||||
'id',
|
||||
'title',
|
||||
'url',
|
||||
'thumbnail',
|
||||
'thumbnail_base')
|
||||
end
|
||||
end
|
||||
@@ -17,8 +17,13 @@ module PostRepr
|
||||
|
||||
module_function
|
||||
|
||||
def base post, current_user = nil
|
||||
json = common(post)
|
||||
def base post, current_user = nil, host: nil
|
||||
json =
|
||||
if host.present?
|
||||
common(post, host:)
|
||||
else
|
||||
common(post)
|
||||
end
|
||||
json['tags'] = tag_json(post)
|
||||
json['uploaded_user'] = post.uploaded_user && UserRepr.base(post.uploaded_user)
|
||||
json['viewed'] = current_user ? current_user.viewed?(post) : false
|
||||
@@ -26,31 +31,59 @@ module PostRepr
|
||||
end
|
||||
|
||||
def detail post, current_user = nil, parent_posts: [], child_posts: [],
|
||||
sibling_posts: { }, related: []
|
||||
base(post, current_user).merge(
|
||||
'parent_posts' => cards(parent_posts),
|
||||
'child_posts' => cards(child_posts),
|
||||
'sibling_posts' => sibling_posts.transform_keys(&:to_s).transform_values { |posts|
|
||||
cards(posts)
|
||||
},
|
||||
'related' => cards(related))
|
||||
sibling_posts: { }, related: [], host: nil
|
||||
if host.present?
|
||||
base(post, current_user, host:).merge(
|
||||
'parent_posts' => cards(parent_posts, host:),
|
||||
'child_posts' => cards(child_posts, host:),
|
||||
'sibling_posts' => sibling_posts.transform_keys(&:to_s).transform_values { |posts|
|
||||
cards(posts, host:)
|
||||
},
|
||||
'related' => cards(related, host:))
|
||||
else
|
||||
base(post, current_user).merge(
|
||||
'parent_posts' => cards(parent_posts),
|
||||
'child_posts' => cards(child_posts),
|
||||
'sibling_posts' => sibling_posts.transform_keys(&:to_s).transform_values { |posts|
|
||||
cards(posts)
|
||||
},
|
||||
'related' => cards(related))
|
||||
end
|
||||
end
|
||||
|
||||
def card post
|
||||
common(post).merge('parent_posts' => [], 'child_posts' => [])
|
||||
def card post, host: nil
|
||||
if host.present?
|
||||
common(post, host:).merge('parent_posts' => [], 'child_posts' => [])
|
||||
else
|
||||
common(post).merge('parent_posts' => [], 'child_posts' => [])
|
||||
end
|
||||
end
|
||||
|
||||
def cards posts
|
||||
posts.map { |post| card(post) }
|
||||
def cards posts, host: nil
|
||||
if host.present?
|
||||
posts.map { |post| card(post, host:) }
|
||||
else
|
||||
posts.map { |post| card(post) }
|
||||
end
|
||||
end
|
||||
|
||||
def many posts, current_user = nil
|
||||
posts.map { |p| base(p, current_user) }
|
||||
def many posts, current_user = nil, host: nil
|
||||
if host.present?
|
||||
posts.map { |p| base(p, current_user, host:) }
|
||||
else
|
||||
posts.map { |p| base(p, current_user) }
|
||||
end
|
||||
end
|
||||
|
||||
def common post
|
||||
def common post, host: nil
|
||||
BASE_FIELDS.to_h { |field| [field.to_s, post.public_send(field)] }
|
||||
.merge('thumbnail' => thumbnail_url(post))
|
||||
.merge(
|
||||
'thumbnail' =>
|
||||
if host.present?
|
||||
thumbnail_url(post, host:)
|
||||
else
|
||||
thumbnail_url(post)
|
||||
end)
|
||||
end
|
||||
|
||||
def tag_json post
|
||||
@@ -65,11 +98,22 @@ module PostRepr
|
||||
}
|
||||
end
|
||||
|
||||
def thumbnail_url post
|
||||
def thumbnail_url post, host: nil
|
||||
return nil unless post.thumbnail.attached?
|
||||
|
||||
Rails.application.routes.url_helpers.rails_blob_url(post.thumbnail, only_path: false)
|
||||
rescue
|
||||
options = { only_path: false }
|
||||
options[:host] = host if host.present?
|
||||
|
||||
Rails.application.routes.url_helpers.rails_storage_proxy_url(post.thumbnail, **options)
|
||||
rescue ActionController::UrlGenerationError, ArgumentError, URI::InvalidURIError => e
|
||||
payload = {
|
||||
post_id: post.id,
|
||||
attachment_id: post.thumbnail.attachment&.id,
|
||||
blob_id: post.thumbnail.blob&.id,
|
||||
error_class: e.class,
|
||||
message: e.message }
|
||||
|
||||
Rails.logger.warn("PostRepr.thumbnail_url failed #{ payload.to_json }")
|
||||
nil
|
||||
end
|
||||
end
|
||||
|
||||
@@ -0,0 +1,202 @@
|
||||
class PostBulkCreator
|
||||
def initialize actor:, posts:, thumbnails:, host: nil
|
||||
@actor_id = actor.id
|
||||
@posts = posts
|
||||
@thumbnails = thumbnails
|
||||
@host = host
|
||||
end
|
||||
|
||||
def run
|
||||
results = Array.new(@posts.length)
|
||||
mutex = Mutex.new
|
||||
next_index = 0
|
||||
|
||||
workers = Array.new(2) do
|
||||
Thread.new do
|
||||
Rails.application.executor.wrap do
|
||||
ActiveRecord::Base.connection_pool.with_connection do
|
||||
actor = User.find(@actor_id)
|
||||
loop do
|
||||
index = nil
|
||||
begin
|
||||
index = mutex.synchronize do
|
||||
current = next_index
|
||||
next_index += 1
|
||||
current
|
||||
end
|
||||
break if index >= @posts.length
|
||||
|
||||
attributes = @posts[index]
|
||||
results[index] = create_row(actor, attributes, index)
|
||||
rescue StandardError => e
|
||||
Rails.logger.error(
|
||||
"post_bulk_creator_worker_failure #{ { error: e.class.name,
|
||||
message: e.message,
|
||||
index: }.to_json }")
|
||||
results[index] = {
|
||||
status: 'failed',
|
||||
recoverable: false,
|
||||
errors: { base: ['登録中にエラーが発生しました．'] },
|
||||
base_errors: [] }
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
workers.each(&:join)
|
||||
|
||||
results.each_index do |index|
|
||||
next if results[index].present?
|
||||
|
||||
results[index] = {
|
||||
status: 'failed',
|
||||
recoverable: false,
|
||||
errors: { base: ['登録中にエラーが発生しました．'] },
|
||||
base_errors: [] }
|
||||
end
|
||||
|
||||
{ results: }
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def create_row actor, attributes, index
|
||||
preflight =
|
||||
PostCreatePreflight.new(
|
||||
attributes: attributes,
|
||||
thumbnail: thumbnail_for(index, attributes),
|
||||
host: @host).run
|
||||
if preflight[:existing_post_id].present?
|
||||
return {
|
||||
status: 'skipped',
|
||||
existing_post_id: preflight[:existing_post_id],
|
||||
existing_post: preflight[:existing_post] }
|
||||
end
|
||||
|
||||
post = PostCreator.new(
|
||||
actor: actor,
|
||||
attributes: normalised_attributes(attributes, preflight, index)).create!
|
||||
result = {
|
||||
status: 'created',
|
||||
post: { id: post.id } }
|
||||
result[:field_warnings] = preflight[:field_warnings] if preflight[:field_warnings].present?
|
||||
result[:base_warnings] = preflight[:base_warnings] if preflight[:base_warnings].present?
|
||||
result
|
||||
rescue PostCreatePreflight::ValidationFailed => e
|
||||
{
|
||||
status: 'failed',
|
||||
recoverable: true,
|
||||
errors: e.fields,
|
||||
base_errors: e.base_errors }
|
||||
rescue ActiveRecord::RecordInvalid => e
|
||||
existing_post = existing_post_for_race(attributes, e.record)
|
||||
if existing_post.present?
|
||||
return {
|
||||
status: 'skipped',
|
||||
existing_post_id: existing_post[:id],
|
||||
existing_post: existing_post }
|
||||
end
|
||||
|
||||
{
|
||||
status: 'failed',
|
||||
recoverable: true,
|
||||
errors: e.record.errors.to_hash,
|
||||
base_errors: e.record.errors[:base] }
|
||||
rescue ActiveRecord::RecordNotUnique => e
|
||||
if e.message.include?('index_posts_on_url')
|
||||
existing_post = existing_post_for_race(attributes)
|
||||
return {
|
||||
status: 'skipped',
|
||||
existing_post_id: existing_post[:id],
|
||||
existing_post: existing_post } if existing_post.present?
|
||||
end
|
||||
|
||||
Rails.logger.error(
|
||||
"post_bulk_creator_record_not_unique #{ { error: e.class.name,
|
||||
message: e.message }.to_json }")
|
||||
{
|
||||
status: 'failed',
|
||||
recoverable: false,
|
||||
errors: { base: ['登録中にエラーが発生しました．'] },
|
||||
base_errors: [] }
|
||||
rescue Tag::NicoTagNormalisationError
|
||||
{
|
||||
status: 'failed',
|
||||
recoverable: true,
|
||||
errors: { tags: ['ニコニコ・タグは直接指定できません．'] },
|
||||
base_errors: [] }
|
||||
rescue Tag::DeprecatedTagNormalisationError
|
||||
{
|
||||
status: 'failed',
|
||||
recoverable: true,
|
||||
errors: { tags: ['廃止済みタグは付与できません．'] },
|
||||
base_errors: [] }
|
||||
rescue PostCreator::VideoMsParseError
|
||||
{
|
||||
status: 'failed',
|
||||
recoverable: true,
|
||||
errors: { video_ms: ['動画時間の記法が不正です．'] },
|
||||
base_errors: [] }
|
||||
rescue Post::RemoteThumbnailFetchFailed
|
||||
{
|
||||
status: 'failed',
|
||||
recoverable: true,
|
||||
errors: { thumbnail_base: ['サムネイル画像の取得に失敗しました．'] },
|
||||
base_errors: [] }
|
||||
rescue ArgumentError => e
|
||||
{
|
||||
status: 'failed',
|
||||
recoverable: true,
|
||||
errors: { base: [e.message] },
|
||||
base_errors: [] }
|
||||
rescue StandardError => e
|
||||
Rails.logger.error(
|
||||
"post_bulk_creator_failure #{ { error: e.class.name,
|
||||
message: e.message }.to_json }")
|
||||
{
|
||||
status: 'failed',
|
||||
recoverable: false,
|
||||
errors: { base: ['登録中にエラーが発生しました．'] },
|
||||
base_errors: [] }
|
||||
end
|
||||
|
||||
def normalised_attributes attributes, preflight, index
|
||||
{
|
||||
url: preflight[:url],
|
||||
title: preflight[:title],
|
||||
thumbnail_base: preflight[:thumbnail_base],
|
||||
thumbnail: thumbnail_for(index, attributes),
|
||||
tags: preflight[:tags],
|
||||
parent_post_ids: preflight[:parent_post_ids],
|
||||
original_created_from: preflight[:original_created_from],
|
||||
original_created_before: preflight[:original_created_before],
|
||||
duration: preflight[:duration],
|
||||
video_ms: preflight[:video_ms],
|
||||
direct_tag_specs: preflight[:direct_tag_specs],
|
||||
default_tag_specs: preflight[:default_tag_specs],
|
||||
snapshot_tag_specs: preflight[:snapshot_tag_specs],
|
||||
post_tag_specs: preflight[:post_tag_specs],
|
||||
tag_sections: preflight[:tag_sections],
|
||||
normalised_parent_post_ids: preflight[:normalised_parent_post_ids] }
|
||||
end
|
||||
|
||||
def thumbnail_for index, attributes
|
||||
return nil if attributes['thumbnail_base'].present? || attributes[:thumbnail_base].present?
|
||||
|
||||
@thumbnails[index]
|
||||
end
|
||||
|
||||
def existing_post_for_race attributes, record = nil
|
||||
return nil if record.present? && !(record.errors.of_kind?(:url, :taken))
|
||||
|
||||
normal_url = PostUrlNormaliser.normalise(attributes['url'] || attributes[:url])
|
||||
return nil if normal_url.blank?
|
||||
|
||||
compact_existing_post(Post.with_attached_thumbnail.find_by(url: normal_url))
|
||||
end
|
||||
|
||||
def compact_existing_post post
|
||||
PostCompactRepr.base(post, host: @host)
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,271 @@
|
||||
class PostCreatePlan
|
||||
VIDEO_TAG_NAME = '動画'.freeze
|
||||
TAGME_TAG_NAME = 'タグ希望'.freeze
|
||||
NO_DEERJIKIST_TAG_NAME = 'ニジラー情報不詳'.freeze
|
||||
|
||||
def initialize attributes:
|
||||
@attributes = attributes.symbolize_keys
|
||||
@existing_tags_by_name = nil
|
||||
end
|
||||
|
||||
def build!
|
||||
direct_tag_specs, tag_sections = parse_direct_tag_specs
|
||||
default_tag_specs = build_default_tag_specs(direct_tag_specs)
|
||||
snapshot_tag_specs = merge_tag_specs(direct_tag_specs + default_tag_specs)
|
||||
preload_existing_tags_by_name!(snapshot_tag_specs.map { _1[:name] })
|
||||
validate_new_tag_specs!(snapshot_tag_specs)
|
||||
post_tag_specs = expand_parent_tag_specs(snapshot_tag_specs)
|
||||
video_ms = normalise_video_ms(snapshot_tag_specs)
|
||||
validate_video_sections!(video_ms, tag_sections)
|
||||
parent_post_ids = normalise_parent_post_ids
|
||||
validate_parent_post_ids!(parent_post_ids)
|
||||
|
||||
{
|
||||
url: @attributes[:url],
|
||||
title: @attributes[:title].to_s,
|
||||
thumbnail_base: @attributes[:thumbnail_base].presence,
|
||||
original_created_from: @attributes[:original_created_from].presence,
|
||||
original_created_before: @attributes[:original_created_before].presence,
|
||||
tags: serialised_tags(direct_tag_specs, tag_sections),
|
||||
display_tags: display_tags(direct_tag_specs, tag_sections),
|
||||
duration: @attributes[:duration].to_s,
|
||||
video_ms: video_ms,
|
||||
parent_post_ids: parent_post_ids.join(' '),
|
||||
direct_tag_specs: direct_tag_specs,
|
||||
default_tag_specs: default_tag_specs,
|
||||
snapshot_tag_specs: snapshot_tag_specs,
|
||||
post_tag_specs: post_tag_specs,
|
||||
tag_sections: tag_sections,
|
||||
normalised_parent_post_ids: parent_post_ids }
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def tag_names = @attributes[:tags].to_s.split
|
||||
|
||||
def parse_direct_tag_specs
|
||||
tag_sections = { }
|
||||
direct_tag_specs = []
|
||||
|
||||
tag_names.each do |raw_name|
|
||||
tag_name, category, sections = parse_raw_tag_name(raw_name)
|
||||
existing_tag = existing_tags_by_name[tag_name]
|
||||
raise Tag::NicoTagNormalisationError if existing_tag&.nico?
|
||||
raise Tag::DeprecatedTagNormalisationError, [existing_tag.name] if existing_tag&.deprecated?
|
||||
|
||||
direct_tag_specs << {
|
||||
name: tag_name,
|
||||
category: (category || existing_tag&.category || 'general').to_sym }
|
||||
if sections.present?
|
||||
tag_sections[tag_name] ||= []
|
||||
tag_sections[tag_name].concat(sections)
|
||||
tag_sections[tag_name] = Tag.merge_section_ranges(tag_sections[tag_name])
|
||||
tag_sections.delete(tag_name) if tag_sections[tag_name] == [[0, nil]]
|
||||
end
|
||||
end
|
||||
|
||||
[merge_tag_specs(direct_tag_specs), tag_sections]
|
||||
end
|
||||
|
||||
def parse_raw_tag_name raw_name
|
||||
name = raw_name.to_s
|
||||
prefix, category =
|
||||
Tag::CATEGORY_PREFIXES.find {
|
||||
name.downcase.start_with?(_1[0])
|
||||
} || ['', nil]
|
||||
name = name.sub(/\A#{ prefix }/i, '')
|
||||
|
||||
sections = []
|
||||
while (match = name.match(/\A(\S*?)\[([^\[\]\s]*)-([^\[\]\s]*)\](\S*)\z/))
|
||||
name = "#{ match[1] }#{ match[4] }"
|
||||
next if match[2].empty? && match[3].empty?
|
||||
|
||||
sections << Tag.normalise_section_range!(
|
||||
begin_raw: match[2],
|
||||
end_raw: match[3],
|
||||
tag_name: name)
|
||||
end
|
||||
if name.include?('[') || name.include?(']')
|
||||
raise Tag::SectionLiteralParseError.new(raw_name, raw_name)
|
||||
end
|
||||
|
||||
[resolved_tag_name(name), category&.to_sym, sections]
|
||||
end
|
||||
|
||||
def build_default_tag_specs direct_tag_specs
|
||||
default_tag_specs = []
|
||||
if direct_tag_specs.length < 10 && direct_tag_specs.none? { _1[:name] == TAGME_TAG_NAME }
|
||||
default_tag_specs << {
|
||||
name: TAGME_TAG_NAME,
|
||||
category: :meta }
|
||||
end
|
||||
if direct_tag_specs.none? { deerjikist_tag_spec?(_1) }
|
||||
default_tag_specs << {
|
||||
name: NO_DEERJIKIST_TAG_NAME,
|
||||
category: :meta }
|
||||
end
|
||||
|
||||
default_tag_specs
|
||||
end
|
||||
|
||||
def validate_new_tag_specs! specs
|
||||
Array(specs).each do |spec|
|
||||
next if existing_tags_by_name.key?(spec[:name])
|
||||
|
||||
validate_new_tag_spec!(spec)
|
||||
end
|
||||
end
|
||||
|
||||
def validate_new_tag_spec! spec
|
||||
tag_name = TagName.new(name: spec[:name])
|
||||
tag = Tag.new(category: spec[:category], tag_name:)
|
||||
return if tag_name.valid? && tag.valid?
|
||||
|
||||
post = Post.new
|
||||
tag_name.errors[:name].each do |message|
|
||||
post.errors.add :tags, "#{ spec[:name] }: #{ message }"
|
||||
end
|
||||
tag.errors.each do |error|
|
||||
next if error.attribute == :tag_name
|
||||
|
||||
post.errors.add :tags, "#{ spec[:name] }: #{ error.message }"
|
||||
end
|
||||
raise ActiveRecord::RecordInvalid, post
|
||||
end
|
||||
|
||||
def expand_parent_tag_specs snapshot_tag_specs
|
||||
existing_snapshot_tags = snapshot_tag_specs.filter_map { existing_tags_by_name[_1[:name]] }
|
||||
expanded_parent_specs =
|
||||
Tag.expand_parent_tags(existing_snapshot_tags)
|
||||
.reject(&:deprecated?)
|
||||
.map { |tag|
|
||||
{
|
||||
name: tag.name,
|
||||
category: tag.category.to_sym } }
|
||||
merge_tag_specs(snapshot_tag_specs + expanded_parent_specs)
|
||||
end
|
||||
|
||||
def merge_tag_specs specs
|
||||
specs.each_with_object({ }) do |spec, merged|
|
||||
merged[spec[:name]] =
|
||||
if merged.key?(spec[:name]) && merged[spec[:name]][:category] != :general
|
||||
merged[spec[:name]]
|
||||
else
|
||||
{
|
||||
name: spec[:name],
|
||||
category: spec[:category] }
|
||||
end
|
||||
end.values.sort_by { _1[:name] }
|
||||
end
|
||||
|
||||
def existing_tags_by_name
|
||||
@existing_tags_by_name ||= begin
|
||||
names = tag_names.map { canonical_tag_name_without_sections(_1) }.uniq
|
||||
Tag.joins(:tag_name).where(tag_names: { name: names }).index_by(&:name)
|
||||
end
|
||||
end
|
||||
|
||||
def preload_existing_tags_by_name! names
|
||||
wanted_names = Array(names).map { _1.to_s }.reject(&:blank?).uniq
|
||||
missing_names = wanted_names - existing_tags_by_name.keys
|
||||
return if missing_names.empty?
|
||||
|
||||
existing_tags_by_name.merge!(
|
||||
Tag.joins(:tag_name)
|
||||
.where(tag_names: { name: missing_names })
|
||||
.index_by(&:name))
|
||||
end
|
||||
|
||||
def canonical_tag_name_without_sections raw_name
|
||||
name, = parse_raw_tag_name(raw_name)
|
||||
name
|
||||
end
|
||||
|
||||
def deerjikist_tag_spec? spec
|
||||
return true if spec[:category] == :deerjikist
|
||||
|
||||
existing_tags_by_name[spec[:name]]&.deerjikist?
|
||||
end
|
||||
|
||||
def normalise_parent_post_ids
|
||||
Array(@attributes[:parent_post_ids]).flat_map { _1.to_s.split }.map { |token|
|
||||
id = Integer(token, exception: false)
|
||||
raise ArgumentError, "親投稿 Id. が不正です: #{ token }" if id.nil? || id <= 0
|
||||
|
||||
id
|
||||
}.uniq.sort
|
||||
end
|
||||
|
||||
def validate_parent_post_ids! ids
|
||||
missing = ids - Post.where(id: ids).pluck(:id)
|
||||
raise ArgumentError, "存在しない親投稿 Id. があります: #{ missing.join(' ') }" if missing.present?
|
||||
end
|
||||
|
||||
def serialised_tags direct_tag_specs, tag_sections
|
||||
direct_tag_specs.map { |spec|
|
||||
"#{ spec[:name] }#{ tag_sections[spec[:name]].to_a.map { section_literal(_1) }.join }"
|
||||
}.sort.join(' ')
|
||||
end
|
||||
|
||||
def display_tags direct_tag_specs, tag_sections
|
||||
direct_tag_specs.map { |spec|
|
||||
{
|
||||
name: spec[:name],
|
||||
category: spec[:category].to_s,
|
||||
section_literals: tag_sections[spec[:name]].to_a.map { section_literal(_1) } }
|
||||
}.sort_by { _1[:name] }
|
||||
end
|
||||
|
||||
def section_literal range
|
||||
begin_ms, end_ms = range
|
||||
"[#{ Post.ms_to_time(begin_ms) }-#{ end_ms ? Post.ms_to_time(end_ms) : '' }]"
|
||||
end
|
||||
|
||||
def normalise_video_ms snapshot_tag_specs
|
||||
return nil unless snapshot_tag_specs.any? { _1[:name] == VIDEO_TAG_NAME }
|
||||
|
||||
video_ms = @attributes[:video_ms]
|
||||
if video_ms.present?
|
||||
value = Integer(video_ms, exception: false)
|
||||
raise PostCreator::VideoMsParseError unless value&.positive?
|
||||
|
||||
return value
|
||||
end
|
||||
|
||||
duration = @attributes[:duration]
|
||||
return nil if duration.blank?
|
||||
|
||||
value = Tag.time_to_ms!(duration.to_s, tag_name: '動画時間')
|
||||
raise PostCreator::VideoMsParseError unless value.positive?
|
||||
|
||||
value
|
||||
rescue Tag::SectionLiteralParseError
|
||||
raise PostCreator::VideoMsParseError
|
||||
end
|
||||
|
||||
def validate_video_sections! video_ms, tag_sections
|
||||
return unless video_ms
|
||||
|
||||
tag_sections.each_value do |ranges|
|
||||
ranges.each do |begin_ms, end_ms|
|
||||
if begin_ms >= video_ms
|
||||
post = Post.new
|
||||
post.errors.add :video_ms, 'タグ区間の開始が動画時間以上です．'
|
||||
raise ActiveRecord::RecordInvalid, post
|
||||
end
|
||||
if end_ms && end_ms > video_ms
|
||||
post = Post.new
|
||||
post.errors.add :video_ms, 'タグ区間の終端が動画時間を超えてゐます．'
|
||||
raise ActiveRecord::RecordInvalid, post
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def resolved_tag_name name
|
||||
tag_name = TagName.includes(:canonical).find_by(name:)
|
||||
return name if tag_name.nil?
|
||||
|
||||
(tag_name.canonical || tag_name).name
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,141 @@
|
||||
class PostCreatePreflight
|
||||
class ValidationFailed < StandardError
|
||||
attr_reader :fields, :base_errors
|
||||
|
||||
def initialize fields: { }, base_errors: []
|
||||
super('入力内容を確認してください．')
|
||||
@fields = fields
|
||||
@base_errors = base_errors
|
||||
end
|
||||
end
|
||||
|
||||
def initialize attributes:, thumbnail: nil, host: nil
|
||||
@attributes = attributes.symbolize_keys
|
||||
@thumbnail = thumbnail
|
||||
@host = host
|
||||
end
|
||||
|
||||
def run
|
||||
preview = PostImportPreviewer.new.preview_rows(
|
||||
rows: [preview_row],
|
||||
fetch_metadata: false).first
|
||||
if preview[:existing_post_id].present?
|
||||
return {
|
||||
url: preview[:url],
|
||||
title: preview[:attributes]['title'],
|
||||
thumbnail_base: preview[:attributes]['thumbnail_base'],
|
||||
tags: preview[:attributes]['tags'],
|
||||
parent_post_ids: preview[:attributes]['parent_post_ids'],
|
||||
original_created_from: preview[:attributes]['original_created_from'],
|
||||
original_created_before: preview[:attributes]['original_created_before'],
|
||||
duration: preview[:attributes]['duration'],
|
||||
video_ms: preview[:attributes]['video_ms'],
|
||||
display_tags: preview[:display_tags] || [],
|
||||
field_warnings: final_field_warnings(preview[:field_warnings] || { }),
|
||||
base_warnings: preview[:base_warnings],
|
||||
existing_post_id: preview[:existing_post_id],
|
||||
existing_post: existing_post_compact(preview[:existing_post_id]) }
|
||||
end
|
||||
|
||||
if preview[:validation_errors].present?
|
||||
raise ValidationFailed.new(fields: preview[:validation_errors])
|
||||
end
|
||||
|
||||
validate_thumbnail_upload!
|
||||
|
||||
plan = PostCreatePlan.new(
|
||||
attributes: {
|
||||
url: preview[:url],
|
||||
title: preview[:attributes]['title'],
|
||||
thumbnail_base: preview[:attributes]['thumbnail_base'],
|
||||
tags: preview[:attributes]['tags'],
|
||||
parent_post_ids: preview[:attributes]['parent_post_ids'],
|
||||
original_created_from: preview[:attributes]['original_created_from'],
|
||||
original_created_before: preview[:attributes]['original_created_before'],
|
||||
duration: preview[:attributes]['duration'],
|
||||
video_ms: preview[:attributes]['video_ms'] }).build!
|
||||
|
||||
{
|
||||
url: plan[:url],
|
||||
title: plan[:title],
|
||||
thumbnail_base: plan[:thumbnail_base],
|
||||
tags: plan[:tags],
|
||||
parent_post_ids: plan[:parent_post_ids],
|
||||
original_created_from: plan[:original_created_from],
|
||||
original_created_before: plan[:original_created_before],
|
||||
duration: plan[:duration],
|
||||
video_ms: plan[:video_ms],
|
||||
display_tags: plan[:display_tags],
|
||||
direct_tag_specs: plan[:direct_tag_specs],
|
||||
default_tag_specs: plan[:default_tag_specs],
|
||||
snapshot_tag_specs: plan[:snapshot_tag_specs],
|
||||
post_tag_specs: plan[:post_tag_specs],
|
||||
tag_sections: plan[:tag_sections],
|
||||
normalised_parent_post_ids: plan[:normalised_parent_post_ids],
|
||||
field_warnings: final_field_warnings(preview[:field_warnings] || { }),
|
||||
base_warnings: preview[:base_warnings],
|
||||
existing_post_id: preview[:existing_post_id],
|
||||
existing_post: existing_post_compact(preview[:existing_post_id]) }
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def preview_row
|
||||
{
|
||||
source_row: 1,
|
||||
url: @attributes[:url].to_s,
|
||||
attributes: {
|
||||
'title' => @attributes[:title].to_s,
|
||||
'thumbnail_base' => @attributes[:thumbnail_base].to_s,
|
||||
'original_created_from' => @attributes[:original_created_from].to_s,
|
||||
'original_created_before' => @attributes[:original_created_before].to_s,
|
||||
'duration' => @attributes[:duration].to_s,
|
||||
'video_ms' => @attributes[:video_ms],
|
||||
'tags' => @attributes[:tags].to_s,
|
||||
'parent_post_ids' => parent_post_ids_text },
|
||||
provenance: {
|
||||
'url' => 'manual',
|
||||
'title' => 'manual',
|
||||
'thumbnail_base' => 'manual',
|
||||
'original_created_from' => 'manual',
|
||||
'original_created_before' => 'manual',
|
||||
'duration' => 'manual',
|
||||
'video_ms' => 'manual',
|
||||
'tags' => 'manual',
|
||||
'parent_post_ids' => 'manual' },
|
||||
tag_sources: {
|
||||
'automatic' => '',
|
||||
'manual' => @attributes[:tags].to_s } }
|
||||
end
|
||||
|
||||
def parent_post_ids_text
|
||||
Array(@attributes[:parent_post_ids]).flat_map { _1.to_s.split }.join(' ')
|
||||
end
|
||||
|
||||
def validate_thumbnail_upload!
|
||||
return if @attributes[:thumbnail_base].present?
|
||||
return if @thumbnail.blank?
|
||||
|
||||
PostThumbnailUploadValidator.validate!(@thumbnail)
|
||||
rescue PostThumbnailUploadValidator::InvalidUpload => e
|
||||
raise ValidationFailed.new(fields: { thumbnail: [e.message] })
|
||||
end
|
||||
|
||||
def existing_post_compact post_id
|
||||
return nil if post_id.blank?
|
||||
|
||||
post = Post.with_attached_thumbnail.find_by(id: post_id)
|
||||
PostCompactRepr.base(post, host: @host)
|
||||
end
|
||||
|
||||
def final_field_warnings field_warnings
|
||||
thumbnail_warnings = (field_warnings['thumbnail_base'] || []).reject { _1 == 'サムネールなし' }
|
||||
if @attributes[:thumbnail_base].blank? && @thumbnail.blank?
|
||||
thumbnail_warnings = (thumbnail_warnings + ['サムネールなし']).uniq
|
||||
end
|
||||
|
||||
next_warnings = field_warnings.except('thumbnail_base')
|
||||
next_warnings['thumbnail_base'] = thumbnail_warnings if thumbnail_warnings.present?
|
||||
next_warnings
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,154 @@
|
||||
class PostCreator
|
||||
class VideoMsParseError < ArgumentError; end
|
||||
|
||||
attr_reader :field_warnings
|
||||
|
||||
def initialize actor:, attributes:
|
||||
@actor = actor
|
||||
@attributes = attributes.symbolize_keys
|
||||
@field_warnings = { }
|
||||
end
|
||||
|
||||
def create!
|
||||
thumbnail_attachment = prepare_thumbnail_attachment
|
||||
post = Post.new(title: @attributes[:title].presence,
|
||||
url: @attributes[:url],
|
||||
thumbnail_base: @attributes[:thumbnail_base].presence,
|
||||
uploaded_user: @actor,
|
||||
original_created_from: @attributes[:original_created_from].presence,
|
||||
original_created_before: @attributes[:original_created_before].presence)
|
||||
|
||||
ApplicationRecord.transaction do
|
||||
post.save!
|
||||
post.thumbnail.attach(thumbnail_attachment) if thumbnail_attachment.present?
|
||||
snapshot_tags = planned_snapshot_tags
|
||||
post_tags = planned_post_tags
|
||||
sections = planned_sections
|
||||
TagVersioning.record_tag_snapshots!(snapshot_tags, created_by_user: @actor)
|
||||
post.video_ms = planned_video_ms
|
||||
post.save!
|
||||
sync_post_tags!(post, post_tags, sections)
|
||||
sync_parent_posts!(post, planned_parent_post_ids)
|
||||
PostVersionRecorder.record!(post:, event_type: :create, created_by_user: @actor)
|
||||
end
|
||||
post
|
||||
rescue StandardError
|
||||
post&.thumbnail&.purge if post&.thumbnail&.attached?
|
||||
raise
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def prepare_thumbnail_attachment
|
||||
PostThumbnailAttachmentBuilder.build(
|
||||
thumbnail: @attributes[:thumbnail],
|
||||
thumbnail_base: @attributes[:thumbnail_base].presence)
|
||||
end
|
||||
|
||||
def planned_snapshot_tags = planned_create_attributes[:snapshot_tags]
|
||||
|
||||
def planned_post_tags = planned_create_attributes[:post_tags]
|
||||
|
||||
def planned_sections = planned_create_attributes[:tag_sections]
|
||||
|
||||
def planned_parent_post_ids = planned_create_attributes[:normalised_parent_post_ids]
|
||||
|
||||
def planned_video_ms
|
||||
planned_create_attributes[:video_ms]
|
||||
end
|
||||
|
||||
def planned_create_attributes
|
||||
@planned_create_attributes ||= begin
|
||||
if @attributes.key?(:snapshot_tag_specs)
|
||||
snapshot_tags = materialise_tags(@attributes[:snapshot_tag_specs] || [])
|
||||
post_tags = materialise_tags(@attributes[:post_tag_specs] || [])
|
||||
{
|
||||
snapshot_tags: snapshot_tags,
|
||||
post_tags: post_tags,
|
||||
tag_sections: materialise_sections(
|
||||
@attributes[:tag_sections] || { },
|
||||
snapshot_tags,
|
||||
post_tags),
|
||||
normalised_parent_post_ids: @attributes[:normalised_parent_post_ids] || [],
|
||||
video_ms: @attributes[:video_ms] }
|
||||
else
|
||||
build_materialised_plan
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def build_materialised_plan
|
||||
plan = PostCreatePlan.new(attributes: @attributes).build!
|
||||
snapshot_tags = materialise_tags(plan[:snapshot_tag_specs] || [])
|
||||
post_tags = materialise_tags(plan[:post_tag_specs] || [])
|
||||
{
|
||||
snapshot_tags: snapshot_tags,
|
||||
post_tags: post_tags,
|
||||
tag_sections: materialise_sections(
|
||||
plan[:tag_sections] || { },
|
||||
snapshot_tags,
|
||||
post_tags),
|
||||
normalised_parent_post_ids: plan[:normalised_parent_post_ids] || [],
|
||||
video_ms: plan[:video_ms] }
|
||||
end
|
||||
|
||||
def materialise_tags specs
|
||||
Array(specs).each_with_object({ }) do |spec, tags|
|
||||
name = spec[:name] || spec['name']
|
||||
category = spec[:category] || spec['category']
|
||||
next if name.blank? || category.blank?
|
||||
|
||||
tag = Tag.find_or_create_by_tag_name!(name, category:)
|
||||
tag.update!(category:) if tag.category.to_sym != category.to_sym
|
||||
tags[name] ||= tag
|
||||
end.values
|
||||
end
|
||||
|
||||
def materialise_sections sections_by_name, snapshot_tags, post_tags
|
||||
tags_by_name = post_tags.index_by(&:name)
|
||||
snapshot_tags.each do |tag|
|
||||
tags_by_name[tag.name] ||= tag
|
||||
end
|
||||
|
||||
sections_by_name.each_with_object({ }) do |(tag_name, ranges), sections|
|
||||
tag = tags_by_name[tag_name.to_s]
|
||||
next if tag.nil?
|
||||
|
||||
sections[tag.id] = Array(ranges).map { |range| [range[0], range[1]] }
|
||||
end
|
||||
end
|
||||
|
||||
def sync_post_tags! post, desired_tags, sections
|
||||
desired_ids = desired_tags.map(&:id).to_set
|
||||
current_ids = post.tags.pluck(:id).to_set
|
||||
Tag.where(id: desired_ids - current_ids).find_each do |tag|
|
||||
PostTag.create_or_find_by!(post:, tag:, created_user: @actor)
|
||||
end
|
||||
PostTagSection.where(post_id: post.id).destroy_all
|
||||
sections.each do |tag_id, ranges|
|
||||
ranges.each do |begin_ms, end_ms|
|
||||
PostTagSection.create!(post_id: post.id,
|
||||
tag_id:,
|
||||
begin_ms:,
|
||||
end_ms:)
|
||||
end
|
||||
end
|
||||
PostTag.where(post_id: post.id,
|
||||
tag_id: (current_ids - desired_ids).to_a).kept.find_each do |post_tag|
|
||||
post_tag.discard_by!(@actor)
|
||||
end
|
||||
end
|
||||
|
||||
def sync_parent_posts! post, ids
|
||||
if ids.include?(post.id)
|
||||
post.errors.add :parent_post_ids, '自分自身を親投稿にはできません．'
|
||||
raise ActiveRecord::RecordInvalid, post
|
||||
end
|
||||
missing = ids - Post.where(id: ids).pluck(:id)
|
||||
if missing.present?
|
||||
post.errors.add :parent_post_ids, "存在しない親投稿 Id. があります: #{ missing.join(' ') }"
|
||||
raise ActiveRecord::RecordInvalid, post
|
||||
end
|
||||
ids.each { |parent_post_id| PostImplication.create_or_find_by!(post:, parent_post_id:) }
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,522 @@
|
||||
require 'time'
|
||||
require 'timeout'
|
||||
|
||||
class PostImportPreviewer
|
||||
FIELDS = [
|
||||
'title',
|
||||
'thumbnail_base',
|
||||
'original_created_from',
|
||||
'original_created_before',
|
||||
'video_ms',
|
||||
'duration',
|
||||
'tags',
|
||||
'parent_post_ids'].freeze
|
||||
FETCH_WARNING_FIELDS = ['url', 'title', 'thumbnail_base'].freeze
|
||||
TITLE_FETCH_WARNING = 'タイトルを取得できませんでした．'.freeze
|
||||
THUMBNAIL_FETCH_WARNING = 'サムネールを取得できませんでした．'.freeze
|
||||
METADATA_FETCH_WARNING = '自動取得に失敗しました．'.freeze
|
||||
|
||||
def preview_rows rows:, fetch_metadata: true, metadata_cache: { }
|
||||
prepared_rows = rows.map { prepare_row(_1) }
|
||||
url_counts = prepared_rows.filter_map { _1[:normal_url] }.tally
|
||||
existing_posts =
|
||||
Post.where(url: prepared_rows.map { _1[:normal_url] }.compact.uniq).index_by(&:url)
|
||||
existing_parent_ids = preload_parent_ids(prepared_rows)
|
||||
preload_metadata!(
|
||||
prepared_rows,
|
||||
fetch_metadata,
|
||||
metadata_cache,
|
||||
existing_posts,
|
||||
url_counts)
|
||||
known_tags =
|
||||
preload_known_tags(
|
||||
prepared_rows,
|
||||
fetch_metadata,
|
||||
metadata_cache,
|
||||
existing_posts,
|
||||
url_counts)
|
||||
prepared_rows.map { |row|
|
||||
preview_row(row,
|
||||
fetch_metadata:,
|
||||
metadata_cache:,
|
||||
existing_posts:,
|
||||
url_counts:,
|
||||
known_tags:,
|
||||
existing_parent_ids:)
|
||||
}
|
||||
end
|
||||
|
||||
def normalised_url value
|
||||
PostUrlNormaliser.normalise(value)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def prepare_row row
|
||||
source = row.symbolize_keys
|
||||
url = source[:url].to_s.strip
|
||||
normal_url = normalised_url(url)
|
||||
source.merge(url_text: url, normal_url:, url_error: validate_url_safety(normal_url))
|
||||
end
|
||||
|
||||
def preview_row row,
|
||||
fetch_metadata:,
|
||||
metadata_cache:,
|
||||
existing_posts:,
|
||||
url_counts:,
|
||||
known_tags:,
|
||||
existing_parent_ids:
|
||||
attributes = initial_attributes(row)
|
||||
provenance = initial_provenance(row)
|
||||
tag_sources = initial_tag_sources(row, attributes, provenance)
|
||||
field_warnings = initial_field_warnings(row)
|
||||
base_warnings = initial_base_warnings(row)
|
||||
url = row[:url_text]
|
||||
provenance['url'] = 'manual'
|
||||
normal_url = row[:normal_url]
|
||||
url_for_metadata = normal_url || url
|
||||
existing_post = normal_url.present? ? existing_posts[normal_url] : nil
|
||||
|
||||
validation_errors = {}
|
||||
validation_errors[:url] = ['URL が不正です．'] if normal_url.blank?
|
||||
if row[:url_error].present?
|
||||
validation_errors[:url] = [row[:url_error]]
|
||||
end
|
||||
if normal_url.present? && url_counts[normal_url].to_i > 1
|
||||
validation_errors[:url] = ['URL が重複しています．']
|
||||
end
|
||||
|
||||
if row[:metadata_url].present? && row[:metadata_url] != url_for_metadata
|
||||
clear_automatic_values!(attributes, provenance, tag_sources)
|
||||
field_warnings = { }
|
||||
base_warnings = [ ]
|
||||
end
|
||||
|
||||
if validation_errors.blank? && normal_url.present? && existing_post
|
||||
attributes['tags'] = merged_tags(tag_sources, provenance['tags'])
|
||||
warnings_present = field_warnings.values.any?(&:present?) || base_warnings.present?
|
||||
return { source_row: row[:source_row],
|
||||
url: normal_url,
|
||||
attributes:,
|
||||
provenance:,
|
||||
tag_sources:,
|
||||
metadata_url: url_for_metadata,
|
||||
skip_reason: 'existing',
|
||||
existing_post_id: existing_post.id,
|
||||
field_warnings:,
|
||||
base_warnings:,
|
||||
validation_errors:,
|
||||
status: warnings_present ? 'warning' : 'ready' }
|
||||
end
|
||||
|
||||
should_fetch = validation_errors.blank?
|
||||
should_fetch &&= should_fetch_metadata?(fetch_metadata, row[:source_row].to_i)
|
||||
if should_fetch
|
||||
clear_fetch_warnings!(field_warnings)
|
||||
metadata = metadata_for(url_for_metadata, metadata_cache)
|
||||
if metadata[:validation_errors].present?
|
||||
validation_errors.merge!(metadata[:validation_errors])
|
||||
else
|
||||
apply_metadata!(attributes, provenance, tag_sources, metadata[:data])
|
||||
apply_fetch_warnings!(field_warnings, metadata[:warnings])
|
||||
end
|
||||
end
|
||||
|
||||
attributes['url'] = url
|
||||
validate_basic_data(attributes, validation_errors)
|
||||
validate_preview_tags(merged_tags(tag_sources, provenance['tags']),
|
||||
validation_errors,
|
||||
known_tags)
|
||||
validate_parents(attributes['parent_post_ids'], validation_errors, existing_parent_ids)
|
||||
attributes.delete('url')
|
||||
attributes['tags'] = merged_tags(tag_sources, provenance['tags'])
|
||||
|
||||
warnings_present = field_warnings.values.any?(&:present?) || base_warnings.present?
|
||||
{ source_row: row[:source_row],
|
||||
url: validation_errors[:url].present? ? url : (normal_url || url),
|
||||
attributes:,
|
||||
provenance:,
|
||||
tag_sources:,
|
||||
metadata_url: url_for_metadata,
|
||||
skip_reason: nil,
|
||||
existing_post_id: nil,
|
||||
field_warnings:,
|
||||
base_warnings:,
|
||||
validation_errors:,
|
||||
status: validation_errors.present? ? 'error' : (warnings_present ? 'warning' : 'ready') }
|
||||
end
|
||||
|
||||
def should_fetch_metadata? fetch_metadata, source_row
|
||||
case fetch_metadata
|
||||
when true then true
|
||||
when Integer then fetch_metadata == source_row
|
||||
when false, nil then false
|
||||
else raise ArgumentError, '取得対象が不正です．'
|
||||
end
|
||||
end
|
||||
|
||||
def initial_attributes row
|
||||
attributes = row[:attributes]&.stringify_keys || { }
|
||||
FIELDS.to_h { |field|
|
||||
value = attributes[field]
|
||||
normalised =
|
||||
if field == 'duration'
|
||||
normalise_duration_attribute(value)
|
||||
elsif field == 'video_ms'
|
||||
value.nil? ? '' : value.to_s
|
||||
else
|
||||
value.to_s
|
||||
end
|
||||
[field, normalised] }
|
||||
end
|
||||
|
||||
def initial_provenance row
|
||||
provenance = row[:provenance]&.stringify_keys || { }
|
||||
FIELDS.to_h { |field| [field, provenance[field].presence || 'automatic'] }
|
||||
.merge('url' => 'manual')
|
||||
end
|
||||
|
||||
def initial_tag_sources row, attributes, provenance
|
||||
sources = row[:tag_sources]&.stringify_keys || { 'automatic' => '', 'manual' => '' }
|
||||
sources['automatic'] = sources['automatic'].to_s
|
||||
sources['manual'] =
|
||||
provenance['tags'] == 'manual' ? attributes['tags'].to_s : sources['manual'].to_s
|
||||
sources
|
||||
end
|
||||
|
||||
def initial_field_warnings row
|
||||
(row[:field_warnings] || { })
|
||||
.stringify_keys
|
||||
.transform_values { |value| Array(value).map(&:to_s) }
|
||||
end
|
||||
|
||||
def initial_base_warnings row
|
||||
Array(row[:base_warnings]).map(&:to_s)
|
||||
end
|
||||
|
||||
def clear_automatic_values! attributes, provenance, tag_sources
|
||||
['title', 'thumbnail_base', 'original_created_from',
|
||||
'original_created_before', 'video_ms', 'duration'].each do |field|
|
||||
attributes[field] = '' if provenance[field] == 'automatic'
|
||||
end
|
||||
tag_sources['automatic'] = ''
|
||||
attributes['tags'] = merged_tags(tag_sources, provenance['tags'])
|
||||
end
|
||||
|
||||
def clear_fetch_warnings! field_warnings
|
||||
FETCH_WARNING_FIELDS.each do |field|
|
||||
field_warnings.delete(field)
|
||||
end
|
||||
end
|
||||
|
||||
def metadata_for url, cache
|
||||
cache[url] ||= fetch_metadata(url)
|
||||
end
|
||||
|
||||
def fetch_metadata url
|
||||
return { data: { }, warnings: { 'url' => ['URL が空です．'] },
|
||||
validation_errors: { } } if url.blank?
|
||||
|
||||
data = sanitise_metadata(PostMetadataFetcher.fetch(url).stringify_keys.compact)
|
||||
warnings = { }
|
||||
add_field_warning!(warnings, 'title', TITLE_FETCH_WARNING) if data['title'].blank?
|
||||
if data['thumbnail_base'].blank?
|
||||
add_field_warning!(warnings, 'thumbnail_base', THUMBNAIL_FETCH_WARNING)
|
||||
end
|
||||
{ data:, warnings:, validation_errors: { } }
|
||||
rescue Preview::UrlSafety::UnsafeUrl => e
|
||||
payload = { error: e.class.name, message: e.message }
|
||||
Rails.logger.info(
|
||||
"post_import_metadata_fetch_unsafe_url #{ payload.to_json }")
|
||||
{ data: { }, warnings: { }, validation_errors: { url: [e.message] } }
|
||||
rescue Preview::HttpFetcher::FetchFailed,
|
||||
Preview::HttpFetcher::ResponseTooLarge => e
|
||||
payload = { error: e.class.name, message: e.message }
|
||||
Rails.logger.info(
|
||||
"post_import_metadata_fetch_failure #{ payload.to_json }")
|
||||
{ data: { }, warnings: { 'url' => [METADATA_FETCH_WARNING] }, validation_errors: { } }
|
||||
end
|
||||
|
||||
def sanitise_metadata metadata
|
||||
{
|
||||
'title' => sanitise_metadata_title(metadata['title']),
|
||||
'thumbnail_base' => sanitise_metadata_url(metadata['thumbnail_base']),
|
||||
'original_created_from' => sanitise_metadata_time(metadata['original_created_from']),
|
||||
'original_created_before' => sanitise_metadata_time(metadata['original_created_before']),
|
||||
'duration' => sanitise_metadata_duration(metadata['duration']),
|
||||
'tags' => metadata['tags'].to_s.presence }.compact
|
||||
end
|
||||
|
||||
def sanitise_metadata_title value
|
||||
value.is_a?(String) ? value.presence : nil
|
||||
end
|
||||
|
||||
def sanitise_metadata_url value
|
||||
return nil unless value.is_a?(String)
|
||||
|
||||
stripped = value.strip
|
||||
return nil if stripped.blank?
|
||||
|
||||
uri = URI.parse(stripped)
|
||||
return nil unless uri.is_a?(URI::HTTP) && uri.host.present?
|
||||
|
||||
stripped
|
||||
rescue URI::InvalidURIError
|
||||
nil
|
||||
end
|
||||
|
||||
def sanitise_metadata_time value
|
||||
return nil unless value.is_a?(String)
|
||||
|
||||
Time.iso8601(value).in_time_zone.change(sec: 0, nsec: 0).iso8601
|
||||
rescue ArgumentError, TypeError
|
||||
nil
|
||||
end
|
||||
|
||||
def sanitise_metadata_duration value
|
||||
return nil unless value.is_a?(String)
|
||||
|
||||
value.presence
|
||||
end
|
||||
|
||||
def normalise_duration_attribute value
|
||||
return '' if value.nil?
|
||||
return value if value.is_a?(String)
|
||||
|
||||
milliseconds = Integer(value, exception: false)
|
||||
return value.to_s if milliseconds.nil? || milliseconds <= 0
|
||||
|
||||
seconds_string = (milliseconds / 1_000.0).to_s
|
||||
seconds_string.end_with?('.0') ? seconds_string.delete_suffix('.0') : seconds_string
|
||||
end
|
||||
|
||||
def preload_metadata! prepared_rows, fetch_metadata, metadata_cache, existing_posts, url_counts
|
||||
urls = prepared_rows.filter_map { |row|
|
||||
next unless row[:normal_url].present?
|
||||
next if row[:url_error].present?
|
||||
next unless should_fetch_metadata?(fetch_metadata, row[:source_row].to_i)
|
||||
next if url_counts[row[:normal_url]].to_i > 1
|
||||
next if existing_posts.key?(row[:normal_url])
|
||||
|
||||
row[:normal_url]
|
||||
}.uniq
|
||||
urls = urls.reject { metadata_cache.key?(_1) }
|
||||
return if urls.empty?
|
||||
|
||||
url_queue = Queue.new
|
||||
result_queue = Queue.new
|
||||
urls.each { url_queue << _1 }
|
||||
workers = [urls.length, 4].min.times.map {
|
||||
Thread.new {
|
||||
Rails.application.executor.wrap do
|
||||
loop do
|
||||
url = url_queue.pop(true)
|
||||
result_queue << [url, safe_fetch_metadata(url)]
|
||||
rescue ThreadError
|
||||
break
|
||||
end
|
||||
end
|
||||
}
|
||||
}
|
||||
Timeout.timeout(15) { workers.each(&:join) }
|
||||
rescue Timeout::Error
|
||||
workers&.each(&:kill)
|
||||
ensure
|
||||
workers&.each(&:join)
|
||||
while result_queue&.size.to_i.positive?
|
||||
url, result = result_queue.pop
|
||||
metadata_cache[url] = result
|
||||
end
|
||||
urls&.each do |url|
|
||||
metadata_cache[url] ||= { data: { }, warnings: { 'url' => [METADATA_FETCH_WARNING] } }
|
||||
end
|
||||
end
|
||||
|
||||
def safe_fetch_metadata url
|
||||
fetch_metadata(url)
|
||||
rescue Preview::UrlSafety::UnsafeUrl => e
|
||||
payload = { error: e.class.name, message: e.message }
|
||||
Rails.logger.info(
|
||||
"post_import_metadata_fetch_unsafe_url #{ payload.to_json }")
|
||||
{ data: { }, warnings: { }, validation_errors: { url: [e.message] } }
|
||||
rescue StandardError => e
|
||||
payload = { error: e.class.name, message: e.message }
|
||||
Rails.logger.error(
|
||||
"post_import_metadata_fetch_unexpected_failure #{ payload.to_json }")
|
||||
{ data: { }, warnings: { 'url' => [METADATA_FETCH_WARNING] }, validation_errors: { } }
|
||||
end
|
||||
|
||||
def preload_known_tags prepared_rows, fetch_metadata, metadata_cache, existing_posts, url_counts
|
||||
names = prepared_rows.flat_map { |row|
|
||||
attributes = initial_attributes(row)
|
||||
provenance = initial_provenance(row)
|
||||
tag_sources = initial_tag_sources(row, attributes, provenance)
|
||||
if metadata_url_changed?(row)
|
||||
clear_automatic_values!(attributes, provenance, tag_sources)
|
||||
end
|
||||
if should_apply_metadata_to_row?(row, fetch_metadata, existing_posts, url_counts)
|
||||
metadata = metadata_for(row[:normal_url], metadata_cache)
|
||||
apply_metadata!(attributes, provenance, tag_sources, metadata[:data])
|
||||
end
|
||||
preview_tag_names(merged_tags(tag_sources, provenance['tags']))
|
||||
}.compact.uniq
|
||||
return { } if names.empty?
|
||||
|
||||
Tag.joins(:tag_name)
|
||||
.where(tag_names: { name: names })
|
||||
.includes(:tag_name)
|
||||
.to_a
|
||||
.index_by(&:name)
|
||||
end
|
||||
|
||||
def preload_parent_ids prepared_rows
|
||||
ids = prepared_rows.flat_map { |row|
|
||||
attributes = initial_attributes(row)
|
||||
preview_parent_ids(attributes['parent_post_ids'])
|
||||
}.uniq
|
||||
return { } if ids.empty?
|
||||
|
||||
Post.where(id: ids).pluck(:id).to_h { [_1, true] }
|
||||
end
|
||||
|
||||
def metadata_url_changed? row
|
||||
row[:metadata_url].present? && row[:metadata_url] != (row[:normal_url] || row[:url_text])
|
||||
end
|
||||
|
||||
def should_apply_metadata_to_row? row, fetch_metadata, existing_posts, url_counts
|
||||
normal_url = row[:normal_url]
|
||||
return false if normal_url.blank?
|
||||
return false if row[:url_error].present?
|
||||
return false if url_counts[normal_url].to_i > 1
|
||||
return false if existing_posts.key?(normal_url)
|
||||
|
||||
should_fetch_metadata?(fetch_metadata, row[:source_row].to_i)
|
||||
end
|
||||
|
||||
def validate_url_safety normal_url
|
||||
return nil if normal_url.blank?
|
||||
|
||||
Preview::UrlSafety.validate(normal_url)
|
||||
nil
|
||||
rescue Preview::UrlSafety::UnsafeUrl => e
|
||||
e.message
|
||||
end
|
||||
|
||||
def apply_metadata! attributes, provenance, tag_sources, metadata
|
||||
metadata.each do |field, value|
|
||||
if field == 'tags'
|
||||
next if provenance['tags'] == 'manual'
|
||||
|
||||
tag_sources['automatic'] = value.to_s
|
||||
attributes['tags'] = merged_tags(tag_sources)
|
||||
next
|
||||
end
|
||||
next unless provenance[field] == 'automatic'
|
||||
|
||||
attributes[field] = value
|
||||
provenance[field] = 'automatic'
|
||||
end
|
||||
end
|
||||
|
||||
def apply_fetch_warnings! field_warnings, warnings
|
||||
warnings.each do |field, values|
|
||||
values.each do |value|
|
||||
add_field_warning!(field_warnings, field, value)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def add_field_warning! field_warnings, field, message
|
||||
field_warnings[field] ||= []
|
||||
field_warnings[field] << message unless field_warnings[field].include?(message)
|
||||
end
|
||||
|
||||
def merged_tags sources, origin = nil
|
||||
return sources['manual'].to_s if origin == 'manual'
|
||||
|
||||
sources['automatic'].to_s
|
||||
end
|
||||
|
||||
def preview_tag_names raw
|
||||
names = raw.to_s.split
|
||||
return [] if names.empty?
|
||||
if names.any? { _1.downcase.start_with?('nico:') }
|
||||
return []
|
||||
end
|
||||
|
||||
names.map { |name| TagName.canonicalise(name.sub(/\[.*\]\z/, '')).first }
|
||||
rescue Tag::SectionLiteralParseError
|
||||
[]
|
||||
end
|
||||
|
||||
def validate_preview_tags raw, errors, known_tags
|
||||
names = raw.to_s.split
|
||||
return if names.empty?
|
||||
if names.any? { _1.downcase.start_with?('nico:') }
|
||||
errors[:tags] = ['ニコニコ・タグは直接指定できません．']
|
||||
return
|
||||
end
|
||||
|
||||
parsed = names.map { |name| TagName.canonicalise(name.sub(/\[.*\]\z/, '')).first }
|
||||
existing = parsed.filter_map { known_tags[_1] }
|
||||
deprecated = existing.select(&:deprecated?).map(&:name)
|
||||
errors[:tags] = ["廃止済みタグがあります: #{ deprecated.join(' ') }"] if deprecated.present?
|
||||
rescue Tag::SectionLiteralParseError
|
||||
errors[:tags] = ['タグ区間の記法が不正です．']
|
||||
end
|
||||
|
||||
def validate_basic_data attributes, errors
|
||||
post = Post.new(title: attributes['title'].presence,
|
||||
url: attributes['url'],
|
||||
thumbnail_base: attributes['thumbnail_base'].presence,
|
||||
original_created_from: attributes['original_created_from'].presence,
|
||||
original_created_before: attributes['original_created_before'].presence,
|
||||
video_ms: parse_video_ms(attributes, errors))
|
||||
post.valid?
|
||||
post.errors.each do |error|
|
||||
next if error.attribute == :url && error.type == :taken
|
||||
|
||||
(errors[error.attribute] ||= []) << error.message
|
||||
end
|
||||
end
|
||||
|
||||
def parse_video_ms attributes, errors
|
||||
return nil unless attributes['tags'].to_s.split.include?('動画')
|
||||
|
||||
video_ms = attributes['video_ms']
|
||||
if video_ms.present?
|
||||
value = Integer(video_ms, exception: false)
|
||||
if value&.positive?
|
||||
return value
|
||||
end
|
||||
|
||||
errors[:video_ms] = ['動画時間の記法が不正です．']
|
||||
return nil
|
||||
end
|
||||
|
||||
value = attributes['duration']
|
||||
return nil if value.blank?
|
||||
|
||||
Tag.time_to_ms!(value.to_s, tag_name: '動画時間')
|
||||
rescue Tag::SectionLiteralParseError
|
||||
errors[:video_ms] = ['動画時間の記法が不正です．']
|
||||
nil
|
||||
end
|
||||
|
||||
def preview_parent_ids raw
|
||||
raw.to_s.split.map { Integer(_1, exception: false) }.compact
|
||||
end
|
||||
|
||||
def validate_parents raw, errors, existing_parent_ids
|
||||
ids = raw.to_s.split.map { Integer(_1, exception: false) }
|
||||
return if ids.compact.empty? && raw.to_s.blank?
|
||||
|
||||
if ids.any? { _1.nil? || _1 <= 0 }
|
||||
errors[:parent_post_ids] = ['親投稿 Id. が不正です．']
|
||||
return
|
||||
end
|
||||
if ids.uniq.any? { !existing_parent_ids[_1] }
|
||||
errors[:parent_post_ids] = ['存在しない親投稿 Id. があります．']
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,198 @@
|
||||
require 'time'
|
||||
require 'date'
|
||||
|
||||
class PostMetadataFetcher
|
||||
TIMESTAMP_PATTERN =
|
||||
/\A(\d{4})-(\d{2})-(\d{2})T(\d{2})
|
||||
(?::(\d{2})(?::(\d{2})(?:\.(\d+))?)?)?
|
||||
(Z|[+-]\d{2}:?\d{2})?\z/x
|
||||
|
||||
def self.fetch raw_url
|
||||
uri, = Preview::UrlSafety.validate(raw_url)
|
||||
response = Preview::HttpFetcher.fetch(
|
||||
uri.to_s,
|
||||
max_bytes: Preview::ThumbnailFetcher::HTML_MAX_BYTES)
|
||||
metadata = Preview::HtmlMetadataExtractor.extract(response)
|
||||
document = Nokogiri::HTML.parse(response.body)
|
||||
content = lambda { |name|
|
||||
document
|
||||
.at_css("meta[property='#{ name }'], meta[name='#{ name }']")
|
||||
&.[]('content')
|
||||
&.strip
|
||||
&.presence
|
||||
}
|
||||
duration = content.call('og:video:duration') || content.call('video:duration')
|
||||
published = content.call('article:published_time') || content.call('date')
|
||||
created_range = original_created_range(published)
|
||||
platform_tags = platform_tags(uri)
|
||||
{ title: metadata[:title],
|
||||
thumbnail_base:
|
||||
Preview::KnownSiteExtractor.thumbnail_url(uri) || metadata[:image_url],
|
||||
original_created_from: serialise_time(created_range&.first),
|
||||
original_created_before: serialise_time(created_range&.last),
|
||||
duration: serialise_duration(duration),
|
||||
display_tags: display_tags(platform_tags),
|
||||
tags: platform_tags.join(' ') }
|
||||
end
|
||||
|
||||
def self.platform_tags uri
|
||||
return ['動画', 'YouTube'] if Preview::KnownSiteExtractor.youtube_video_id(uri)
|
||||
return ['動画', 'ニコニコ'] if Preview::KnownSiteExtractor.niconico_video_id(uri)
|
||||
|
||||
[]
|
||||
end
|
||||
|
||||
def self.display_tags names
|
||||
return [] if names.blank?
|
||||
|
||||
existing_tags =
|
||||
Tag
|
||||
.joins(:tag_name)
|
||||
.where(tag_names: { name: names })
|
||||
.index_by(&:name)
|
||||
|
||||
names.map { |name|
|
||||
tag = existing_tags[name]
|
||||
{
|
||||
name: name,
|
||||
category: (tag&.category || 'meta'),
|
||||
section_literals: [] }
|
||||
}
|
||||
end
|
||||
|
||||
def self.original_created_range value
|
||||
return nil if value.blank?
|
||||
|
||||
raw = value.to_s.strip
|
||||
from, before =
|
||||
case raw
|
||||
when /\A(\d{4})\z/
|
||||
year = Regexp.last_match(1).to_i
|
||||
from = Time.zone.local(year, 1, 1)
|
||||
[from, from + 1.year]
|
||||
when /\A(\d{4})-(\d{2})\z/
|
||||
year = Regexp.last_match(1).to_i
|
||||
month = Regexp.last_match(2).to_i
|
||||
return nil unless month.between?(1, 12)
|
||||
|
||||
from = Time.zone.local(year, month, 1)
|
||||
[from, from + 1.month]
|
||||
when /\A(\d{4})-(\d{2})-(\d{2})\z/
|
||||
year = Regexp.last_match(1).to_i
|
||||
month = Regexp.last_match(2).to_i
|
||||
day = Regexp.last_match(3).to_i
|
||||
return nil unless Date.valid_date?(year, month, day)
|
||||
|
||||
from = Time.zone.local(year, month, day)
|
||||
[from, from + 1.day]
|
||||
else
|
||||
parse_timestamp_range(raw)
|
||||
end
|
||||
return nil if from.nil? || before.nil?
|
||||
|
||||
[from, before]
|
||||
rescue ArgumentError, TypeError
|
||||
nil
|
||||
end
|
||||
|
||||
def self.parse_timestamp_range raw
|
||||
match = raw.match(TIMESTAMP_PATTERN)
|
||||
return nil unless match
|
||||
|
||||
year = match[1].to_i
|
||||
month = match[2].to_i
|
||||
day = match[3].to_i
|
||||
hour = match[4].to_i
|
||||
minute = match[5]&.to_i || 0
|
||||
second = match[6]&.to_i || 0
|
||||
fraction = match[7]
|
||||
offset = match[8]
|
||||
return nil unless valid_timestamp_components?(year, month, day, hour, minute, second)
|
||||
return nil unless valid_offset?(offset)
|
||||
|
||||
nanoseconds = parse_nanoseconds(fraction)
|
||||
timestamp =
|
||||
if offset.present?
|
||||
Time.new(
|
||||
year,
|
||||
month,
|
||||
day,
|
||||
hour,
|
||||
minute,
|
||||
second + Rational(nanoseconds, 1_000_000_000),
|
||||
parse_offset(offset)).in_time_zone
|
||||
else
|
||||
Time.zone.local(year, month, day, hour, minute, second).change(nsec: nanoseconds)
|
||||
end
|
||||
|
||||
from = timestamp.change(sec: 0, nsec: 0)
|
||||
before =
|
||||
if match[5].nil?
|
||||
from + 1.hour
|
||||
else
|
||||
from + 1.minute
|
||||
end
|
||||
[from, before]
|
||||
end
|
||||
|
||||
def self.parse_nanoseconds value
|
||||
return 0 if value.blank?
|
||||
|
||||
digits = value[0, 9].ljust(9, '0')
|
||||
Integer(digits, 10)
|
||||
end
|
||||
|
||||
def self.parse_offset value
|
||||
return '+00:00' if value == 'Z'
|
||||
|
||||
value.match?(/\A[+-]\d{2}:\d{2}\z/) ? value : "#{ value[0, 3] }:#{ value[3, 2] }"
|
||||
end
|
||||
|
||||
def self.serialise_duration value
|
||||
seconds = Float(value)
|
||||
return nil unless seconds.positive?
|
||||
|
||||
milliseconds = (seconds * 1_000).round
|
||||
seconds_string = (milliseconds / 1_000.0).to_s
|
||||
seconds_string.end_with?('.0') ? seconds_string.delete_suffix('.0') : seconds_string
|
||||
rescue ArgumentError, TypeError
|
||||
nil
|
||||
end
|
||||
|
||||
def self.valid_timestamp_components? year, month, day, hour, minute, second
|
||||
return false unless Date.valid_date?(year, month, day)
|
||||
return false unless hour.between?(0, 23)
|
||||
return false unless minute.between?(0, 59)
|
||||
return false unless second.between?(0, 59)
|
||||
|
||||
true
|
||||
end
|
||||
|
||||
def self.valid_offset? value
|
||||
return true if value.nil? || value == 'Z'
|
||||
|
||||
match = value.match(/\A([+-])(\d{2}):?(\d{2})\z/)
|
||||
return false if match.nil?
|
||||
|
||||
hours = match[2].to_i
|
||||
minutes = match[3].to_i
|
||||
hours.between?(0, 23) && minutes.between?(0, 59)
|
||||
end
|
||||
|
||||
def self.serialise_time value
|
||||
return nil if value.nil?
|
||||
|
||||
value.nsec.zero? ? value.iso8601 : value.iso8601(9)
|
||||
end
|
||||
|
||||
private_class_method :platform_tags,
|
||||
:display_tags,
|
||||
:original_created_range,
|
||||
:parse_timestamp_range,
|
||||
:parse_nanoseconds,
|
||||
:parse_offset,
|
||||
:serialise_duration,
|
||||
:valid_timestamp_components?,
|
||||
:valid_offset?,
|
||||
:serialise_time
|
||||
end
|
||||
@@ -0,0 +1,11 @@
|
||||
class PostThumbnailAttachmentBuilder
|
||||
def self.build thumbnail:, thumbnail_base:
|
||||
if thumbnail_base.present?
|
||||
return Post.remote_thumbnail_attachment(thumbnail_base)
|
||||
end
|
||||
|
||||
return nil if thumbnail.blank?
|
||||
|
||||
Post.resized_thumbnail_attachment(thumbnail)
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,34 @@
|
||||
class PostThumbnailUploadValidator
|
||||
MAX_THUMBNAIL_BYTES = 20 * 1024 * 1024
|
||||
ALLOWED_CONTENT_TYPES = Preview::ThumbnailFetcher::RASTER_IMAGE_CONTENT_TYPES.freeze
|
||||
|
||||
class InvalidUpload < StandardError; end
|
||||
|
||||
def self.validate! thumbnail
|
||||
return if thumbnail.blank?
|
||||
unless thumbnail.is_a?(ActionDispatch::Http::UploadedFile)
|
||||
raise InvalidUpload, 'thumbnail upload が不正です．'
|
||||
end
|
||||
raise InvalidUpload, 'thumbnail file size が大きすぎます．' if thumbnail.size > MAX_THUMBNAIL_BYTES
|
||||
raise InvalidUpload, 'サムネイル画像の形式が不正です．' unless allowed_content_type?(thumbnail.content_type)
|
||||
bytes = thumbnail.read
|
||||
raise InvalidUpload, 'サムネイル画像の形式が不正です．' if Post.svg_document_bytes?(bytes)
|
||||
unless Post.raster_thumbnail_bytes?(bytes)
|
||||
raise InvalidUpload, 'サムネイル画像の形式が不正です．'
|
||||
end
|
||||
|
||||
attachment = Post.resized_thumbnail_attachment(
|
||||
StringIO.new(bytes),
|
||||
content_type: thumbnail.content_type)
|
||||
attachment[:io].close if attachment[:io].respond_to?(:close)
|
||||
rescue MiniMagick::Error, Timeout::Error
|
||||
raise InvalidUpload, 'サムネイル画像の変換に失敗しました．'
|
||||
ensure
|
||||
thumbnail&.rewind if thumbnail.respond_to?(:rewind)
|
||||
end
|
||||
|
||||
def self.allowed_content_type? content_type
|
||||
mime_type = content_type.to_s.split(';', 2).first.to_s.downcase.strip
|
||||
ALLOWED_CONTENT_TYPES.include?(mime_type)
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,13 @@
|
||||
class PostUrlNormaliser
|
||||
def self.normalise raw_url
|
||||
value = raw_url.to_s.strip
|
||||
uri = URI.parse(value)
|
||||
return nil unless uri.is_a?(URI::HTTP) && uri.host.present?
|
||||
|
||||
uri.host = uri.host.downcase
|
||||
uri.path = uri.path.sub(/\/\z/, '') if uri.path.present?
|
||||
PostUrlSanitisationRule.sanitise(uri.to_s)
|
||||
rescue URI::InvalidURIError
|
||||
nil
|
||||
end
|
||||
end
|
||||
@@ -67,7 +67,8 @@ module Preview
|
||||
log_failure(:timeout, url: uri&.to_s || raw_url, error: e.class.name, message: e.message)
|
||||
raise FetchTimeout, e.message
|
||||
rescue SocketError, SystemCallError, OpenSSL::SSL::SSLError, EOFError => e
|
||||
log_failure(:network_error, url: uri&.to_s || raw_url, error: e.class.name, message: e.message)
|
||||
log_failure(:network_error, url: uri&.to_s || raw_url, error: e.class.name,
|
||||
message: e.message)
|
||||
raise FetchFailed, e.message
|
||||
end
|
||||
|
||||
|
||||
@@ -4,6 +4,15 @@ module Preview
|
||||
youtube_thumbnail(uri)
|
||||
end
|
||||
|
||||
def self.youtube_video_id(uri)
|
||||
case uri.host&.downcase
|
||||
when 'youtu.be'
|
||||
uri.path.split('/').reject(&:blank?).first
|
||||
when 'www.youtube.com', 'youtube.com', 'm.youtube.com'
|
||||
uri.path == '/watch' ? URI.decode_www_form(uri.query.to_s).to_h['v'] : nil
|
||||
end&.then { _1 if _1.match?(/\A[A-Za-z0-9_-]{6,20}\z/) }
|
||||
end
|
||||
|
||||
def self.niconico_video_id(uri)
|
||||
case uri.host&.downcase
|
||||
when 'www.nicovideo.jp', 'nicovideo.jp'
|
||||
@@ -14,14 +23,8 @@ module Preview
|
||||
end
|
||||
|
||||
def self.youtube_thumbnail(uri)
|
||||
id =
|
||||
case uri.host&.downcase
|
||||
when 'youtu.be'
|
||||
uri.path.split('/').reject(&:blank?).first
|
||||
when 'www.youtube.com', 'youtube.com', 'm.youtube.com'
|
||||
uri.path == '/watch' ? URI.decode_www_form(uri.query.to_s).to_h['v'] : nil
|
||||
end
|
||||
return unless id&.match?(/\A[A-Za-z0-9_-]{6,20}\z/)
|
||||
id = youtube_video_id(uri)
|
||||
return unless id
|
||||
|
||||
"https://i.ytimg.com/vi/#{ id }/hqdefault.jpg"
|
||||
end
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
module Preview
|
||||
class ThumbnailFetcher
|
||||
class GenerationFailed < StandardError; end
|
||||
ALLOWED_IMAGE_CONTENT_TYPES = [
|
||||
RASTER_IMAGE_CONTENT_TYPES = [
|
||||
'image/jpeg', 'image/png', 'image/gif', 'image/webp'
|
||||
].freeze
|
||||
HTML_MAX_BYTES = 1.megabyte
|
||||
@@ -25,6 +25,24 @@ module Preview
|
||||
fetch_image!(metadata[:image_url])
|
||||
end
|
||||
|
||||
def self.fetch_image_response(raw_url)
|
||||
uri, = UrlSafety.validate(raw_url)
|
||||
response = HttpFetcher.fetch(uri.to_s)
|
||||
unless Post.remote_thumbnail_image_bytes?(
|
||||
response.body,
|
||||
content_type: response.content_type)
|
||||
raise GenerationFailed, 'サムネール画像が見つかりませんでした．'
|
||||
end
|
||||
|
||||
response
|
||||
rescue HttpFetcher::FetchTimeout
|
||||
raise
|
||||
rescue HttpFetcher::ResponseTooLarge
|
||||
raise
|
||||
rescue HttpFetcher::FetchFailed
|
||||
raise GenerationFailed, 'サムネール画像を取得できませんでした．'
|
||||
end
|
||||
|
||||
def self.title(raw_url)
|
||||
uri, = UrlSafety.validate(raw_url)
|
||||
HtmlMetadataExtractor.extract(
|
||||
@@ -35,7 +53,9 @@ module Preview
|
||||
return nil if url.blank?
|
||||
|
||||
response = HttpFetcher.fetch(url)
|
||||
return nil unless allowed_image_content_type?(response.content_type)
|
||||
return nil unless Post.remote_thumbnail_image_bytes?(
|
||||
response.body,
|
||||
content_type: response.content_type)
|
||||
|
||||
response.body
|
||||
rescue HttpFetcher::FetchTimeout
|
||||
@@ -45,18 +65,7 @@ module Preview
|
||||
end
|
||||
|
||||
def self.fetch_image!(url)
|
||||
response = HttpFetcher.fetch(url)
|
||||
unless allowed_image_content_type?(response.content_type)
|
||||
raise GenerationFailed, 'サムネール画像が見つかりませんでした．'
|
||||
end
|
||||
|
||||
response.body
|
||||
rescue HttpFetcher::FetchTimeout
|
||||
raise
|
||||
rescue HttpFetcher::ResponseTooLarge
|
||||
raise
|
||||
rescue HttpFetcher::FetchFailed
|
||||
raise GenerationFailed, 'サムネール画像を取得できませんでした．'
|
||||
fetch_image_response(url).body
|
||||
end
|
||||
|
||||
def self.niconico_thumbnail_url(uri)
|
||||
@@ -70,26 +79,24 @@ module Preview
|
||||
|
||||
xml.at_xpath('//thumbnail_url')&.text&.strip.presence
|
||||
rescue HttpFetcher::FetchFailed, HttpFetcher::FetchTimeout => e
|
||||
Rails.logger.info("preview_niconico_getthumbinfo_fallback #{ { url: uri.to_s,
|
||||
video_id:,
|
||||
error: e.class.name,
|
||||
message: e.message }.to_json }")
|
||||
payload = {
|
||||
url: uri.to_s,
|
||||
video_id:,
|
||||
error: e.class.name,
|
||||
message: e.message }
|
||||
Rails.logger.info("preview_niconico_getthumbinfo_fallback #{ payload.to_json }")
|
||||
nil
|
||||
rescue Nokogiri::XML::SyntaxError => e
|
||||
Rails.logger.info("preview_niconico_getthumbinfo_fallback #{ { url: uri.to_s,
|
||||
video_id:,
|
||||
error: e.class.name,
|
||||
message: e.message }.to_json }")
|
||||
payload = {
|
||||
url: uri.to_s,
|
||||
video_id:,
|
||||
error: e.class.name,
|
||||
message: e.message }
|
||||
Rails.logger.info("preview_niconico_getthumbinfo_fallback #{ payload.to_json }")
|
||||
nil
|
||||
end
|
||||
|
||||
def self.allowed_image_content_type?(content_type)
|
||||
mime_type = content_type.to_s.split(';', 2).first.downcase.strip
|
||||
ALLOWED_IMAGE_CONTENT_TYPES.include?(mime_type)
|
||||
end
|
||||
|
||||
private_class_method :fetch_image_or_nil, :fetch_image!,
|
||||
:niconico_thumbnail_url,
|
||||
:allowed_image_content_type?
|
||||
:niconico_thumbnail_url
|
||||
end
|
||||
end
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
require 'open-uri'
|
||||
require 'set'
|
||||
require 'time'
|
||||
|
||||
@@ -127,12 +126,12 @@ module Youtube
|
||||
return if post.thumbnail.attached?
|
||||
return if thumbnail_url.blank?
|
||||
|
||||
post.thumbnail.attach(
|
||||
io: URI.open(thumbnail_url),
|
||||
filename: File.basename(URI.parse(thumbnail_url).path),
|
||||
content_type: 'image/jpeg')
|
||||
|
||||
post.resized_thumbnail!
|
||||
post.attach_thumbnail_from_url!(thumbnail_url)
|
||||
rescue Post::RemoteThumbnailFetchFailed => e
|
||||
Rails.logger.info("youtube_sync_thumbnail_fetch_failed #{ { post_id: post.id,
|
||||
thumbnail_url:,
|
||||
error: e.class.name,
|
||||
message: e.message }.to_json }")
|
||||
end
|
||||
|
||||
def youtube_url_regexp id
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
require "active_support/core_ext/integer/time"
|
||||
require 'active_support/core_ext/integer/time'
|
||||
|
||||
Rails.application.configure do
|
||||
# Settings specified here will take precedence over those in config/application.rb.
|
||||
@@ -17,8 +17,8 @@ Rails.application.configure do
|
||||
|
||||
# Enable/disable Action Controller caching. By default Action Controller caching is disabled.
|
||||
# Run rails dev:cache to toggle Action Controller caching.
|
||||
if Rails.root.join("tmp/caching-dev.txt").exist?
|
||||
config.public_file_server.headers = { "cache-control" => "public, max-age=#{2.days.to_i}" }
|
||||
if Rails.root.join('tmp/caching-dev.txt').exist?
|
||||
config.public_file_server.headers = { 'cache-control' => "public, max-age=#{2.days.to_i}" }
|
||||
else
|
||||
config.action_controller.perform_caching = false
|
||||
end
|
||||
@@ -36,7 +36,11 @@ Rails.application.configure do
|
||||
config.action_mailer.perform_caching = false
|
||||
|
||||
# Set localhost to be used by links generated in mailer templates.
|
||||
config.action_mailer.default_url_options = { host: "localhost", port: 3000 }
|
||||
config.action_mailer.default_url_options = { host: 'localhost', port: 3000 }
|
||||
Rails.application.routes.default_url_options.merge!(
|
||||
host: 'localhost',
|
||||
port: 3002,
|
||||
protocol: 'http')
|
||||
|
||||
# Print deprecation notices to the Rails logger.
|
||||
config.active_support.deprecation = :log
|
||||
|
||||
@@ -52,6 +52,8 @@ Rails.application.routes.draw do
|
||||
|
||||
resources :posts, only: [:index, :show, :create, :update] do
|
||||
collection do
|
||||
get :metadata
|
||||
post :bulk
|
||||
get :random
|
||||
get :changes
|
||||
get :versions, to: 'post_versions#index'
|
||||
|
||||
@@ -70,11 +70,17 @@ namespace :nico do
|
||||
unless post.thumbnail.attached?
|
||||
thumbnail_base = fetch_thumbnail.(post.url) rescue nil
|
||||
if thumbnail_base.present?
|
||||
post.thumbnail.attach(
|
||||
io: URI.open(thumbnail_base),
|
||||
filename: File.basename(URI.parse(thumbnail_base).path),
|
||||
content_type: 'image/jpeg')
|
||||
attrs[:thumbnail_base] = thumbnail_base
|
||||
begin
|
||||
post.attach_thumbnail_from_url!(thumbnail_base)
|
||||
rescue Post::RemoteThumbnailFetchFailed => e
|
||||
payload = {
|
||||
post_id: post.id,
|
||||
thumbnail_base:,
|
||||
error: e.class.name,
|
||||
message: e.message }
|
||||
Rails.logger.info("nico_sync_thumbnail_fetch_failed #{ payload.to_json }")
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
@@ -82,7 +88,6 @@ namespace :nico do
|
||||
post_changed = post.changed?
|
||||
if post_changed
|
||||
post.save!
|
||||
post.resized_thumbnail! if post.thumbnail.attached?
|
||||
end
|
||||
else
|
||||
post_created = true
|
||||
@@ -91,13 +96,18 @@ namespace :nico do
|
||||
post = Post.new(title:, url:, thumbnail_base:, uploaded_user: nil,
|
||||
original_created_from:, original_created_before:)
|
||||
if thumbnail_base.present?
|
||||
post.thumbnail.attach(
|
||||
io: URI.open(thumbnail_base),
|
||||
filename: File.basename(URI.parse(thumbnail_base).path),
|
||||
content_type: 'image/jpeg')
|
||||
begin
|
||||
post.attach_thumbnail_from_url!(thumbnail_base)
|
||||
rescue Post::RemoteThumbnailFetchFailed => e
|
||||
payload = {
|
||||
post_id: nil,
|
||||
thumbnail_base:,
|
||||
error: e.class.name,
|
||||
message: e.message }
|
||||
Rails.logger.info("nico_sync_thumbnail_fetch_failed #{ payload.to_json }")
|
||||
end
|
||||
end
|
||||
post.save!
|
||||
post.resized_thumbnail!
|
||||
sync_post_tags!(post, [Tag.tagme.id, Tag.bot.id, Tag.niconico.id, Tag.video.id])
|
||||
end
|
||||
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
require 'rails_helper'
|
||||
require 'tempfile'
|
||||
|
||||
RSpec.describe Post, type: :model do
|
||||
before do
|
||||
@@ -65,4 +66,296 @@ RSpec.describe Post, type: :model do
|
||||
expect(post.errors.details.fetch(:url)).to include(error: :taken, value: post.url)
|
||||
end
|
||||
end
|
||||
|
||||
describe 'thumbnail processing' do
|
||||
def image_blob(width:, height:, background:, draw: nil)
|
||||
Tempfile.create(['post-thumbnail', '.png']) do |file|
|
||||
MiniMagick::Tool::Convert.new do |convert|
|
||||
convert.size "#{ width }x#{ height }"
|
||||
convert.xc background
|
||||
draw&.call(convert)
|
||||
convert << file.path
|
||||
end
|
||||
File.binread(file.path)
|
||||
end
|
||||
end
|
||||
|
||||
def upload_for(blob)
|
||||
StringIO.new(blob).tap(&:rewind)
|
||||
end
|
||||
|
||||
def read_image(attachment)
|
||||
blob =
|
||||
attachment.is_a?(Hash) ? attachment.fetch(:io).read : attachment.download
|
||||
MiniMagick::Image.read(blob)
|
||||
end
|
||||
|
||||
def colour_at(image, x, y)
|
||||
image.get_pixels.fetch(y).fetch(x)
|
||||
end
|
||||
|
||||
def expect_green(pixel)
|
||||
expect(pixel[1]).to be > pixel[0] + 40
|
||||
expect(pixel[1]).to be > pixel[2] + 40
|
||||
end
|
||||
|
||||
def expect_red(pixel)
|
||||
expect(pixel[0]).to be > pixel[1] + 40
|
||||
expect(pixel[0]).to be > pixel[2] + 40
|
||||
end
|
||||
|
||||
def expect_blue(pixel)
|
||||
expect(pixel[2]).to be > pixel[0] + 40
|
||||
expect(pixel[2]).to be > pixel[1] + 40
|
||||
end
|
||||
|
||||
describe '.resized_thumbnail_attachment' do
|
||||
it 'fits a wide image within 180x180 without distorting it' do
|
||||
blob = image_blob(
|
||||
width: 360,
|
||||
height: 180,
|
||||
background: 'red',
|
||||
draw: -> convert {
|
||||
convert.fill 'green'
|
||||
convert.draw 'rectangle 90,0 269,179'
|
||||
})
|
||||
|
||||
resized = described_class.resized_thumbnail_attachment(upload_for(blob))
|
||||
image = read_image(resized)
|
||||
|
||||
expect(image.dimensions).to eq([180, 90])
|
||||
expect_red(colour_at(image, 0, 45))
|
||||
expect_green(colour_at(image, 90, 45))
|
||||
expect_red(colour_at(image, 179, 45))
|
||||
end
|
||||
|
||||
it 'fits a tall image within 180x180 without distorting it' do
|
||||
blob = image_blob(
|
||||
width: 180,
|
||||
height: 360,
|
||||
background: 'red',
|
||||
draw: -> convert {
|
||||
convert.fill 'green'
|
||||
convert.draw 'rectangle 0,90 179,269'
|
||||
})
|
||||
|
||||
resized = described_class.resized_thumbnail_attachment(upload_for(blob))
|
||||
image = read_image(resized)
|
||||
|
||||
expect(image.dimensions).to eq([90, 180])
|
||||
expect_red(colour_at(image, 45, 0))
|
||||
expect_green(colour_at(image, 45, 90))
|
||||
expect_red(colour_at(image, 45, 179))
|
||||
end
|
||||
|
||||
it 'keeps a square image square without distortion' do
|
||||
blob = image_blob(
|
||||
width: 180,
|
||||
height: 180,
|
||||
background: 'red',
|
||||
draw: -> convert {
|
||||
convert.fill 'blue'
|
||||
convert.draw 'rectangle 90,0 179,179'
|
||||
})
|
||||
|
||||
resized = described_class.resized_thumbnail_attachment(upload_for(blob))
|
||||
image = read_image(resized)
|
||||
|
||||
expect(image.dimensions).to eq([180, 180])
|
||||
expect_red(colour_at(image, 20, 90))
|
||||
expect_blue(colour_at(image, 160, 90))
|
||||
end
|
||||
end
|
||||
|
||||
describe '#attach_thumbnail_from_url!' do
|
||||
it 'attaches a fetched remote image through the common resize path' do
|
||||
blob = image_blob(
|
||||
width: 240,
|
||||
height: 180,
|
||||
background: 'red',
|
||||
draw: -> convert {
|
||||
convert.fill 'green'
|
||||
convert.draw 'rectangle 30,0 209,179'
|
||||
})
|
||||
response = Preview::HttpFetcher::Response.new(
|
||||
blob,
|
||||
'image/png',
|
||||
'https://example.com/thumb.png')
|
||||
allow(Preview::ThumbnailFetcher).to receive(:fetch_image_response)
|
||||
.with('https://example.com/thumb.png')
|
||||
.and_return(response)
|
||||
|
||||
post = described_class.create!(title: 'title', url: 'https://example.com/post')
|
||||
|
||||
expect(post.thumbnail).to receive(:attach).once.and_call_original
|
||||
post.attach_thumbnail_from_url!('https://example.com/thumb.png')
|
||||
|
||||
expect(post.thumbnail).to be_attached
|
||||
image = read_image(post.thumbnail)
|
||||
expect(image.dimensions).to eq([180, 135])
|
||||
end
|
||||
|
||||
it 'does not attach anything when thumbnail conversion fails' do
|
||||
response = Preview::HttpFetcher::Response.new(
|
||||
'not-an-image',
|
||||
'image/png',
|
||||
'https://example.com/thumb.png')
|
||||
allow(Preview::ThumbnailFetcher).to receive(:fetch_image_response)
|
||||
.with('https://example.com/thumb.png')
|
||||
.and_return(response)
|
||||
allow(described_class).to receive(:resized_thumbnail_attachment)
|
||||
.and_raise(MiniMagick::Error, 'convert failed')
|
||||
|
||||
post = described_class.create!(title: 'title', url: 'https://example.com/post')
|
||||
|
||||
expect {
|
||||
post.attach_thumbnail_from_url!('https://example.com/thumb.png')
|
||||
}.to raise_error(Post::RemoteThumbnailFetchFailed, 'convert failed')
|
||||
|
||||
expect(post.thumbnail).not_to be_attached
|
||||
end
|
||||
|
||||
it 'keeps an existing thumbnail when remote conversion fails' do
|
||||
existing = described_class.create!(title: 'title', url: 'https://example.com/post')
|
||||
existing.thumbnail.attach(
|
||||
io: StringIO.new('existing'),
|
||||
filename: 'existing.jpg',
|
||||
content_type: 'image/jpeg')
|
||||
blob_id = existing.thumbnail.blob.id
|
||||
response = Preview::HttpFetcher::Response.new(
|
||||
'not-an-image',
|
||||
'image/png',
|
||||
'https://example.com/thumb.png')
|
||||
allow(Preview::ThumbnailFetcher).to receive(:fetch_image_response)
|
||||
.with('https://example.com/thumb.png')
|
||||
.and_return(response)
|
||||
allow(described_class).to receive(:resized_thumbnail_attachment)
|
||||
.and_raise(MiniMagick::Error, 'convert failed')
|
||||
|
||||
expect {
|
||||
existing.attach_thumbnail_from_url!('https://example.com/thumb.png')
|
||||
}.to raise_error(Post::RemoteThumbnailFetchFailed, 'convert failed')
|
||||
|
||||
existing.reload
|
||||
expect(existing.thumbnail).to be_attached
|
||||
expect(existing.thumbnail.blob.id).to eq(blob_id)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
describe 'original created datetime validation' do
|
||||
it 'allows unrelated updates on persisted posts with second-bearing datetimes' do
|
||||
post = described_class.create!(title: 'title', url: 'https://example.com/post')
|
||||
post.update_columns(
|
||||
original_created_from: Time.zone.parse('2024-01-01T12:34:30Z'),
|
||||
original_created_before: Time.zone.parse('2024-01-01T12:35:30Z')
|
||||
)
|
||||
|
||||
post.title = 'updated title'
|
||||
|
||||
expect(post).to be_valid
|
||||
expect { post.save! }.not_to raise_error
|
||||
end
|
||||
|
||||
it 'rejects second-bearing updates when the datetime field changes' do
|
||||
post = described_class.create!(title: 'title', url: 'https://example.com/post')
|
||||
|
||||
post.original_created_from = '2024-01-01T12:34:30Z'
|
||||
|
||||
expect(post).to be_invalid
|
||||
expect(post.errors[:original_created_from]).to eq(
|
||||
[described_class::ORIGINAL_CREATED_MINUTE_PRECISION_MESSAGE]
|
||||
)
|
||||
end
|
||||
|
||||
it 'accepts fixing persisted datetimes to minute precision' do
|
||||
post = described_class.create!(title: 'title', url: 'https://example.com/post')
|
||||
post.update_columns(
|
||||
original_created_from: Time.zone.parse('2024-01-01T12:34:30Z'),
|
||||
original_created_before: Time.zone.parse('2024-01-01T12:35:30Z')
|
||||
)
|
||||
|
||||
post.original_created_from = '2024-01-01T12:34Z'
|
||||
post.original_created_before = '2024-01-01T12:35Z'
|
||||
|
||||
expect(post).to be_valid
|
||||
expect { post.save! }.not_to raise_error
|
||||
end
|
||||
|
||||
it 'adds only the minute-precision error for second precision values' do
|
||||
post = described_class.new(
|
||||
title: 'title',
|
||||
url: 'https://example.com/post',
|
||||
original_created_from: '2024-01-01T12:34:30',
|
||||
original_created_before: '2024-01-01T12:35'
|
||||
)
|
||||
|
||||
expect(post).to be_invalid
|
||||
expect(post.errors[:original_created_from]).to eq(
|
||||
[described_class::ORIGINAL_CREATED_MINUTE_PRECISION_MESSAGE]
|
||||
)
|
||||
expect(post.errors[:original_created_at]).to be_empty
|
||||
end
|
||||
|
||||
it 'adds only the minute-precision error for fractional-second values' do
|
||||
post = described_class.new(
|
||||
title: 'title',
|
||||
url: 'https://example.com/post',
|
||||
original_created_from: '2024-01-01T12:34:00.123',
|
||||
original_created_before: '2024-01-01T12:35'
|
||||
)
|
||||
|
||||
expect(post).to be_invalid
|
||||
expect(post.errors[:original_created_from]).to eq(
|
||||
[described_class::ORIGINAL_CREATED_MINUTE_PRECISION_MESSAGE]
|
||||
)
|
||||
expect(post.errors[:original_created_at]).to be_empty
|
||||
end
|
||||
|
||||
it 'checks range rules only for valid minute-precision endpoints' do
|
||||
post = described_class.new(
|
||||
title: 'title',
|
||||
url: 'https://example.com/post',
|
||||
original_created_from: '2024-01-01T12:34',
|
||||
original_created_before: '2024-01-01T12:34'
|
||||
)
|
||||
|
||||
expect(post).to be_invalid
|
||||
expect(post.errors[:original_created_at]).to eq(
|
||||
[described_class::ORIGINAL_CREATED_ORDER_MESSAGE]
|
||||
)
|
||||
end
|
||||
|
||||
it 'accepts a one-minute range at minute precision' do
|
||||
post = described_class.new(
|
||||
title: 'title',
|
||||
url: 'https://example.com/post',
|
||||
original_created_from: '2024-01-01T12:34',
|
||||
original_created_before: '2024-01-01T12:35'
|
||||
)
|
||||
|
||||
expect(post).to be_valid
|
||||
end
|
||||
|
||||
it 'rejects invalid calendar dates and invalid hours' do
|
||||
invalid_dates = [
|
||||
'2024-02-31T12:00',
|
||||
'2023-02-29T12:00',
|
||||
'2024-02-29T24:00'
|
||||
]
|
||||
|
||||
invalid_dates.each do |value|
|
||||
post = described_class.new(
|
||||
title: 'title',
|
||||
url: 'https://example.com/post',
|
||||
original_created_from: value
|
||||
)
|
||||
|
||||
expect(post).to be_invalid
|
||||
expect(post.errors[:original_created_from]).to eq(
|
||||
[described_class::ORIGINAL_CREATED_INVALID_MESSAGE]
|
||||
)
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
@@ -9,11 +9,11 @@ RSpec.describe 'Posts API', type: :request do
|
||||
# resized_thumbnail! が MiniMagick 依存でコケやすいので request spec ではスタブしとくのが無難。
|
||||
before do
|
||||
allow_any_instance_of(Post).to receive(:resized_thumbnail!).and_return(true)
|
||||
allow(Post).to receive(:resized_thumbnail_attachment).and_return(
|
||||
io: StringIO.new('dummy'),
|
||||
filename: 'resized_thumbnail.jpg',
|
||||
content_type: 'image/jpeg'
|
||||
)
|
||||
allow(Post).to receive(:resized_thumbnail_attachment) do
|
||||
{ io: StringIO.new('dummy'),
|
||||
filename: 'resized_thumbnail.jpg',
|
||||
content_type: 'image/jpeg' }
|
||||
end
|
||||
end
|
||||
|
||||
def create_nico_tag!(name)
|
||||
@@ -21,8 +21,7 @@ RSpec.describe 'Posts API', type: :request do
|
||||
end
|
||||
|
||||
def dummy_upload
|
||||
# 中身は何でもいい（加工処理はスタブしてる）
|
||||
Rack::Test::UploadedFile.new(StringIO.new('dummy'), 'image/jpeg', original_filename: 'dummy.jpg')
|
||||
real_thumbnail_upload
|
||||
end
|
||||
|
||||
def real_thumbnail_upload
|
||||
@@ -102,34 +101,34 @@ RSpec.describe 'Posts API', type: :request do
|
||||
end
|
||||
end
|
||||
|
||||
describe "GET /posts" do
|
||||
describe 'GET /posts' do
|
||||
let!(:user) { create_member_user! }
|
||||
|
||||
let!(:tag_name) { TagName.create!(name: "spec_tag") }
|
||||
let!(:tag_name) { TagName.create!(name: 'spec_tag') }
|
||||
let!(:tag) { Tag.create!(tag_name:, category: :general) }
|
||||
let!(:tag_name2) { TagName.create!(name: 'unko') }
|
||||
let!(:tag2) { Tag.create!(tag_name: tag_name2, category: :deerjikist) }
|
||||
let!(:alias_tag_name) { TagName.create!(name: 'manko', canonical: tag_name) }
|
||||
|
||||
let!(:hit_post) do
|
||||
Post.create!(uploaded_user: user, title: "hello spec world",
|
||||
Post.create!(uploaded_user: user, title: 'hello spec world',
|
||||
url: 'https://example.com/spec2').tap do |p|
|
||||
PostTag.create!(post: p, tag:)
|
||||
end
|
||||
end
|
||||
|
||||
let!(:miss_post) do
|
||||
Post.create!(uploaded_user: user, title: "unrelated title",
|
||||
Post.create!(uploaded_user: user, title: 'unrelated title',
|
||||
url: 'https://example.com/spec3').tap do |p|
|
||||
PostTag.create!(post: p, tag: tag2)
|
||||
end
|
||||
end
|
||||
|
||||
it "returns posts with tag name in JSON" do
|
||||
get "/posts"
|
||||
it 'returns posts with tag name in JSON' do
|
||||
get '/posts'
|
||||
|
||||
expect(response).to have_http_status(:ok)
|
||||
posts = json.fetch("posts")
|
||||
posts = json.fetch('posts')
|
||||
|
||||
# 全postの全tagが name を含むこと
|
||||
expect(posts).not_to be_empty
|
||||
@@ -142,8 +141,8 @@ RSpec.describe 'Posts API', type: :request do
|
||||
expect(json['count']).to be_an(Integer)
|
||||
|
||||
# spec_tag を含む投稿が存在すること
|
||||
all_tag_names = posts.flat_map { |p| p["tags"].map { |t| t["name"] } }
|
||||
expect(all_tag_names).to include("spec_tag")
|
||||
all_tag_names = posts.flat_map { |p| p['tags'].map { |t| t['name'] } }
|
||||
expect(all_tag_names).to include('spec_tag')
|
||||
end
|
||||
|
||||
it 'keeps children and sections keys in non-detail tag responses' do
|
||||
@@ -162,9 +161,9 @@ RSpec.describe 'Posts API', type: :request do
|
||||
])
|
||||
end
|
||||
|
||||
context "when q is provided" do
|
||||
it "filters posts by q (hit case)" do
|
||||
get "/posts", params: { tags: "spec_tag" }
|
||||
context 'when q is provided' do
|
||||
it 'filters posts by q (hit case)' do
|
||||
get '/posts', params: { tags: 'spec_tag' }
|
||||
|
||||
expect(response).to have_http_status(:ok)
|
||||
posts = json.fetch('posts')
|
||||
@@ -182,8 +181,8 @@ RSpec.describe 'Posts API', type: :request do
|
||||
end
|
||||
end
|
||||
|
||||
it "filters posts by q (hit case by alias)" do
|
||||
get "/posts", params: { tags: "manko" }
|
||||
it 'filters posts by q (hit case by alias)' do
|
||||
get '/posts', params: { tags: 'manko' }
|
||||
|
||||
expect(response).to have_http_status(:ok)
|
||||
posts = json.fetch('posts')
|
||||
@@ -201,11 +200,11 @@ RSpec.describe 'Posts API', type: :request do
|
||||
end
|
||||
end
|
||||
|
||||
it "returns empty posts when nothing matches" do
|
||||
get "/posts", params: { tags: "no_such_keyword_12345" }
|
||||
it 'returns empty posts when nothing matches' do
|
||||
get '/posts', params: { tags: 'no_such_keyword_12345' }
|
||||
|
||||
expect(response).to have_http_status(:ok)
|
||||
expect(json.fetch("posts")).to eq([])
|
||||
expect(json.fetch('posts')).to eq([])
|
||||
expect(json.fetch('count')).to eq(0)
|
||||
end
|
||||
end
|
||||
@@ -693,6 +692,73 @@ RSpec.describe 'Posts API', type: :request do
|
||||
end
|
||||
end
|
||||
|
||||
describe 'GET /posts/metadata' do
|
||||
let(:member) { create(:user, :member) }
|
||||
|
||||
it 'returns compact existing post data without fetching external metadata' do
|
||||
sign_in_as(member)
|
||||
existing = create(
|
||||
:post,
|
||||
title: 'existing post',
|
||||
url: 'https://example.com/existing')
|
||||
existing.thumbnail.attach(
|
||||
io: StringIO.new('thumbnail'),
|
||||
filename: 'thumbnail.jpg',
|
||||
content_type: 'image/jpeg')
|
||||
expect(PostMetadataFetcher).not_to receive(:fetch)
|
||||
|
||||
get '/posts/metadata', params: { url: 'https://example.com/existing' }
|
||||
|
||||
expect(response).to have_http_status(:ok)
|
||||
expect(json).to include(
|
||||
'url' => existing.url,
|
||||
'existing_post_id' => existing.id,
|
||||
'field_warnings' => { })
|
||||
expect(json.fetch('existing_post')).to include(
|
||||
'id' => existing.id,
|
||||
'title' => existing.title,
|
||||
'url' => existing.url,
|
||||
'thumbnail_base' => existing.thumbnail_base)
|
||||
expect(json.dig('existing_post', 'thumbnail'))
|
||||
.to include('/rails/active_storage/blobs/proxy/')
|
||||
end
|
||||
|
||||
it 'returns fetched metadata and structured display tags' do
|
||||
sign_in_as(member)
|
||||
allow(Preview::UrlSafety).to receive(:validate)
|
||||
allow(PostMetadataFetcher).to receive(:fetch).and_return(
|
||||
title: 'fetched title',
|
||||
thumbnail_base: 'https://example.com/thumbnail.jpg',
|
||||
tags: 'character:虹夏',
|
||||
display_tags: [{ name: '虹夏', category: 'character' }],
|
||||
original_created_from: nil,
|
||||
original_created_before: nil,
|
||||
duration: '1:00',
|
||||
video_ms: 60_000)
|
||||
|
||||
get '/posts/metadata', params: { url: 'https://example.com/new' }
|
||||
|
||||
expect(response).to have_http_status(:ok)
|
||||
expect(json).to include(
|
||||
'url' => 'https://example.com/new',
|
||||
'title' => 'fetched title',
|
||||
'duration' => '1:00',
|
||||
'video_ms' => 60_000,
|
||||
'field_warnings' => { })
|
||||
expect(json.fetch('display_tags')).to eq(
|
||||
[{ 'name' => '虹夏', 'category' => 'character' }])
|
||||
end
|
||||
|
||||
it 'returns URL validation errors as 422' do
|
||||
sign_in_as(member)
|
||||
|
||||
get '/posts/metadata', params: { url: 'file:///etc/passwd' }
|
||||
|
||||
expect(response).to have_http_status(:unprocessable_entity)
|
||||
expect(json.fetch('errors')).to have_key('url')
|
||||
end
|
||||
end
|
||||
|
||||
describe 'POST /posts' do
|
||||
let(:member) { create(:user, :member) }
|
||||
let!(:alias_tag_name) { TagName.create!(name: 'manko', canonical: tag_name) }
|
||||
@@ -712,6 +778,46 @@ RSpec.describe 'Posts API', type: :request do
|
||||
expect(response).to have_http_status(:forbidden)
|
||||
end
|
||||
|
||||
it 'dry-runs without persisting posts or new tags' do
|
||||
sign_in_as(member)
|
||||
counts = [Post.count, Tag.count, TagName.count]
|
||||
|
||||
post '/posts?dry=1', params: post_write_params(
|
||||
title: 'dry-run post',
|
||||
url: 'https://example.com/dry-run',
|
||||
tags: 'character:new_dry_run_tag')
|
||||
|
||||
expect(response).to have_http_status(:ok)
|
||||
expect(json).to include(
|
||||
'url' => 'https://example.com/dry-run',
|
||||
'tags' => 'new_dry_run_tag',
|
||||
'existing_post_id' => nil)
|
||||
expect(json.fetch('display_tags')).to eq(
|
||||
[{ 'name' => 'new_dry_run_tag',
|
||||
'category' => 'character',
|
||||
'section_literals' => [] }])
|
||||
expect([Post.count, Tag.count, TagName.count]).to eq(counts)
|
||||
end
|
||||
|
||||
it 'dry-runs an existing URL without validating its upload' do
|
||||
sign_in_as(member)
|
||||
existing = create(:post, url: 'https://example.com/dry-existing')
|
||||
invalid_upload = Rack::Test::UploadedFile.new(
|
||||
StringIO.new('<svg></svg>'),
|
||||
'image/png',
|
||||
original_filename: 'thumbnail.png')
|
||||
|
||||
post '/posts?dry=1', params: post_write_params(
|
||||
title: '',
|
||||
url: existing.url,
|
||||
tags: '',
|
||||
thumbnail: invalid_upload)
|
||||
|
||||
expect(response).to have_http_status(:ok)
|
||||
expect(json).to include('existing_post_id' => existing.id)
|
||||
expect(json.fetch('field_warnings')).not_to have_key('thumbnail_base')
|
||||
end
|
||||
|
||||
it '201 and creates post + tags when member' do
|
||||
sign_in_as(member)
|
||||
|
||||
@@ -769,7 +875,8 @@ RSpec.describe 'Posts API', type: :request do
|
||||
)
|
||||
|
||||
expect(response).to have_http_status(:created)
|
||||
expect(open_transactions).to eq([baseline_open_transactions])
|
||||
expect(open_transactions).to eq(
|
||||
[baseline_open_transactions, baseline_open_transactions])
|
||||
end
|
||||
|
||||
it 'returns 422 and does not create a post when thumbnail resize fails' do
|
||||
@@ -827,7 +934,7 @@ RSpec.describe 'Posts API', type: :request do
|
||||
|
||||
expect(response).to have_http_status(:unprocessable_entity)
|
||||
expect(json.fetch('errors')).to include(
|
||||
'tags' => ['廃止済みタグは付与できません．']
|
||||
'tags' => ['廃止済みタグがあります: deprecated_direct_tag']
|
||||
)
|
||||
end
|
||||
|
||||
@@ -1054,7 +1161,7 @@ RSpec.describe 'Posts API', type: :request do
|
||||
)
|
||||
end
|
||||
|
||||
context "when nico tag already exists in tags" do
|
||||
context 'when nico tag already exists in tags' do
|
||||
before do
|
||||
Tag.find_undiscard_or_create_by!(
|
||||
tag_name: TagName.find_undiscard_or_create_by!(name: 'nico:nico_tag'),
|
||||
@@ -1251,6 +1358,47 @@ RSpec.describe 'Posts API', type: :request do
|
||||
end
|
||||
end
|
||||
|
||||
describe 'POST /posts/bulk' do
|
||||
let(:member) { create(:user, :member) }
|
||||
|
||||
it 'parses the manifest and indexed thumbnail parts' do
|
||||
sign_in_as(member)
|
||||
manifest = [{ 'url' => 'https://example.com/bulk',
|
||||
'title' => 'bulk post',
|
||||
'tags' => 'spec_tag',
|
||||
'parent_post_ids' => '' }]
|
||||
creator = instance_double(
|
||||
PostBulkCreator,
|
||||
run: { results: [{ status: 'created', post: { id: 123 } }] })
|
||||
allow(PostBulkCreator).to receive(:new).and_return(creator)
|
||||
|
||||
post '/posts/bulk', params: {
|
||||
posts: JSON.generate(manifest),
|
||||
thumbnails: { '0' => real_thumbnail_upload } }
|
||||
|
||||
expect(response).to have_http_status(:ok)
|
||||
expect(json.fetch('results')).to eq(
|
||||
[{ 'status' => 'created', 'post' => { 'id' => 123 } }])
|
||||
expect(PostBulkCreator).to have_received(:new) do |arguments|
|
||||
expect(arguments[:actor]).to eq(member)
|
||||
expect(arguments[:posts]).to eq(manifest)
|
||||
expect(arguments[:thumbnails].keys).to eq([0])
|
||||
expect(arguments[:host]).to eq('http://www.example.com')
|
||||
end
|
||||
end
|
||||
|
||||
it 'rejects malformed manifests as a request-level error' do
|
||||
sign_in_as(member)
|
||||
|
||||
post '/posts/bulk', params: {
|
||||
posts: '{',
|
||||
thumbnails: { '0' => real_thumbnail_upload } }
|
||||
|
||||
expect(response).to have_http_status(:bad_request)
|
||||
expect(json.fetch('message')).to eq('posts manifest の JSON が不正です．')
|
||||
end
|
||||
end
|
||||
|
||||
describe 'PUT /posts/:id' do
|
||||
let(:member) { create(:user, :member) }
|
||||
|
||||
@@ -1307,7 +1455,7 @@ RSpec.describe 'Posts API', type: :request do
|
||||
)
|
||||
end
|
||||
|
||||
context "when nico tag already exists in tags" do
|
||||
context 'when nico tag already exists in tags' do
|
||||
before do
|
||||
Tag.find_undiscard_or_create_by!(
|
||||
tag_name: TagName.find_undiscard_or_create_by!(name: 'nico:nico_tag'),
|
||||
@@ -1564,7 +1712,7 @@ RSpec.describe 'Posts API', type: :request do
|
||||
expect(post_record.reload.title).to eq('updated by other user')
|
||||
end
|
||||
|
||||
it 'returns 409 with mergeable true when stale tag changes do not conflict but merge is not requested' do
|
||||
it 'returns mergeable 409 for stale non-conflicting tag changes without merge' do
|
||||
sign_in_as(member)
|
||||
|
||||
base_version = create_post_version_for!(post_record.reload)
|
||||
@@ -2206,6 +2354,91 @@ RSpec.describe 'Posts API', type: :request do
|
||||
|
||||
expect(response).to have_http_status(:unprocessable_entity)
|
||||
end
|
||||
|
||||
it 'rejects second-bearing original created timestamps on POST /posts' do
|
||||
sign_in_as(member)
|
||||
|
||||
post '/posts', params: post_write_params(
|
||||
title: 'invalid original created from',
|
||||
url: 'https://example.com/invalid-original-created-from',
|
||||
tags: 'spec_tag',
|
||||
thumbnail: dummy_upload,
|
||||
original_created_from: '2020-01-01T00:00:01Z')
|
||||
|
||||
expect(response).to have_http_status(:unprocessable_entity)
|
||||
expect(json.fetch('errors')).to include(
|
||||
'original_created_from' => ['オリジナルの作成日時は分単位で入力してください．']
|
||||
)
|
||||
expect(json.fetch('errors')).not_to have_key('original_created_at')
|
||||
end
|
||||
|
||||
it 'rejects fractional-second original created timestamps on POST /posts' do
|
||||
sign_in_as(member)
|
||||
|
||||
post '/posts', params: post_write_params(
|
||||
title: 'invalid original created before',
|
||||
url: 'https://example.com/invalid-original-created-before',
|
||||
tags: 'spec_tag',
|
||||
thumbnail: dummy_upload,
|
||||
original_created_before: '2020-01-01T00:00:00.123Z')
|
||||
|
||||
expect(response).to have_http_status(:unprocessable_entity)
|
||||
expect(json.fetch('errors')).to include(
|
||||
'original_created_before' => ['オリジナルの作成日時は分単位で入力してください．']
|
||||
)
|
||||
expect(json.fetch('errors')).not_to have_key('original_created_at')
|
||||
end
|
||||
|
||||
it 'rejects non-increasing original created ranges on POST /posts' do
|
||||
sign_in_as(member)
|
||||
|
||||
post '/posts', params: post_write_params(
|
||||
title: 'non-increasing original created range',
|
||||
url: 'https://example.com/non-increasing-original-created-range',
|
||||
tags: 'spec_tag',
|
||||
thumbnail: dummy_upload,
|
||||
original_created_from: '2020-01-01T00:00Z',
|
||||
original_created_before: '2020-01-01T00:00Z')
|
||||
|
||||
expect(response).to have_http_status(:unprocessable_entity)
|
||||
expect(json.fetch('errors')).to include(
|
||||
'original_created_at' => ['オリジナルの作成日時の順番がをかしぃです．']
|
||||
)
|
||||
end
|
||||
|
||||
it 'accepts original created ranges that are exactly one minute on POST /posts' do
|
||||
sign_in_as(member)
|
||||
|
||||
post '/posts', params: post_write_params(
|
||||
title: 'valid original created range',
|
||||
url: 'https://example.com/valid-original-created-range',
|
||||
tags: 'spec_tag',
|
||||
thumbnail: dummy_upload,
|
||||
original_created_from: '2020-01-01T00:00Z',
|
||||
original_created_before: '2020-01-01T00:01Z')
|
||||
|
||||
expect(response).to have_http_status(:created)
|
||||
expect(Time.iso8601(json.fetch('original_created_from')))
|
||||
.to eq(Time.utc(2020, 1, 1, 0, 0))
|
||||
expect(Time.iso8601(json.fetch('original_created_before')))
|
||||
.to eq(Time.utc(2020, 1, 1, 0, 1))
|
||||
end
|
||||
|
||||
it 'rejects unparseable original created timestamps on PUT /posts/:id' do
|
||||
sign_in_as(member)
|
||||
base_version = create_post_version_for!(post_record)
|
||||
|
||||
put "/posts/#{post_record.id}", params: post_write_params(
|
||||
base_version_no: base_version.version_no,
|
||||
title: 'updated title',
|
||||
tags: 'spec_tag',
|
||||
original_created_from: 'not-a-time')
|
||||
|
||||
expect(response).to have_http_status(:unprocessable_entity)
|
||||
expect(json.fetch('errors')).to include(
|
||||
'original_created_from' => ['オリジナルの作成日時の形式が不正です．']
|
||||
)
|
||||
end
|
||||
end
|
||||
|
||||
describe 'tag versioning from post write actions' do
|
||||
|
||||
@@ -1,27 +0,0 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe 'Wiki body search', type: :request do
|
||||
let!(:user) { create_member_user! }
|
||||
|
||||
it 'searches wiki pages by body text' do
|
||||
pending '#336 で対応予定'
|
||||
|
||||
Wiki::Commit.create_content!(
|
||||
tag_name: TagName.create!(name: 'wiki_body_search_hit'),
|
||||
body: 'unique body keyword for wiki search',
|
||||
created_by_user: user,
|
||||
message: 'init')
|
||||
|
||||
Wiki::Commit.create_content!(
|
||||
tag_name: TagName.create!(name: 'wiki_body_search_miss'),
|
||||
body: 'ordinary body',
|
||||
created_by_user: user,
|
||||
message: 'init')
|
||||
|
||||
get '/wiki/search', params: { body: 'unique body keyword' }
|
||||
|
||||
expect(response).to have_http_status(:ok)
|
||||
expect(json.map { |page| page['title'] }).to include('wiki_body_search_hit')
|
||||
expect(json.map { |page| page['title'] }).not_to include('wiki_body_search_miss')
|
||||
end
|
||||
end
|
||||
@@ -1,37 +0,0 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe 'Wiki restore', type: :request do
|
||||
let!(:user) { create_member_user! }
|
||||
|
||||
def auth_headers user
|
||||
{ 'X-Transfer-Code' => user.inheritance_code }
|
||||
end
|
||||
|
||||
it 'restores wiki page to previous version' do
|
||||
pending '#337 で対応予定'
|
||||
|
||||
page =
|
||||
Wiki::Commit.create_content!(
|
||||
tag_name: TagName.create!(name: 'wiki_restore_page'),
|
||||
body: 'v1',
|
||||
created_by_user: user,
|
||||
message: 'init')
|
||||
|
||||
v1 = page.wiki_versions.order(:version_no).last
|
||||
|
||||
Wiki::Commit.content!(
|
||||
page:,
|
||||
body: 'v2',
|
||||
created_user: user,
|
||||
message: 'edit',
|
||||
base_revision_id: page.current_revision.id)
|
||||
|
||||
post "/wiki/#{ page.id }/restore",
|
||||
params: { version_no: v1.version_no },
|
||||
headers: auth_headers(user)
|
||||
|
||||
expect(response).to have_http_status(:ok)
|
||||
expect(page.reload.body).to eq('v1')
|
||||
expect(page.wiki_versions.order(:version_no).last.event_type).to eq('restore')
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,59 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe PostBulkCreator do
|
||||
it 'limits workers to two and keeps failures in their request slots' do
|
||||
actor = instance_double(User, id: 123)
|
||||
allow(User).to receive(:find).with(123) {
|
||||
instance_double(User, id: 123)
|
||||
}
|
||||
mutex = Mutex.new
|
||||
active = 0
|
||||
maximum_active = 0
|
||||
|
||||
allow(PostCreatePreflight).to receive(:new) do |attributes:, **|
|
||||
preflight = instance_double(PostCreatePreflight)
|
||||
allow(preflight).to receive(:run) do
|
||||
mutex.synchronize do
|
||||
active += 1
|
||||
maximum_active = [maximum_active, active].max
|
||||
end
|
||||
sleep 0.02
|
||||
mutex.synchronize { active -= 1 }
|
||||
attributes.symbolize_keys.merge(
|
||||
existing_post_id: nil,
|
||||
field_warnings: { },
|
||||
base_warnings: [])
|
||||
end
|
||||
preflight
|
||||
end
|
||||
allow(PostCreator).to receive(:new) do |attributes:, **|
|
||||
creator = instance_double(PostCreator)
|
||||
if attributes[:title] == 'broken'
|
||||
allow(creator).to receive(:create!).and_raise(StandardError, 'broken')
|
||||
else
|
||||
post = instance_double(Post, id: attributes[:title].delete_prefix('post ').to_i)
|
||||
allow(creator).to receive(:create!).and_return(post)
|
||||
end
|
||||
creator
|
||||
end
|
||||
posts = [
|
||||
{ 'title' => 'post 1', 'url' => 'https://example.com/1' },
|
||||
{ 'title' => 'broken', 'url' => 'https://example.com/2' },
|
||||
{ 'title' => 'post 3', 'url' => 'https://example.com/3' },
|
||||
{ 'title' => 'post 4', 'url' => 'https://example.com/4' }]
|
||||
|
||||
results = described_class.new(
|
||||
actor:,
|
||||
posts:,
|
||||
thumbnails: { }).run.fetch(:results)
|
||||
|
||||
expect(maximum_active).to eq(2)
|
||||
expect(results.length).to eq(posts.length)
|
||||
expect(results.map { _1[:status] }).to eq(
|
||||
['created', 'failed', 'created', 'created'])
|
||||
expect(results[0].dig(:post, :id)).to eq(1)
|
||||
expect(results[1]).to include(status: 'failed', recoverable: false)
|
||||
expect(results[2].dig(:post, :id)).to eq(3)
|
||||
expect(results[3].dig(:post, :id)).to eq(4)
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,82 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe PostCreatePlan do
|
||||
def create_tag! name, category
|
||||
Tag.create!(name:, category:)
|
||||
end
|
||||
|
||||
before do
|
||||
create_tag!('タグ希望', :meta)
|
||||
create_tag!('ニジラー情報不詳', :meta)
|
||||
end
|
||||
|
||||
it 'plans direct and existing default tags without persisting records' do
|
||||
counts = [TagName.count, Tag.count]
|
||||
|
||||
plan = described_class.new(
|
||||
attributes: {
|
||||
url: 'https://example.com/post',
|
||||
title: 'title',
|
||||
tags: 'character:new_character',
|
||||
parent_post_ids: '' }).build!
|
||||
|
||||
expect(plan[:tags]).to eq('new_character')
|
||||
expect(plan[:direct_tag_specs]).to eq(
|
||||
[{ name: 'new_character', category: :character }])
|
||||
expect(plan[:default_tag_specs]).to include(
|
||||
{ name: 'タグ希望', category: :meta },
|
||||
{ name: 'ニジラー情報不詳', category: :meta })
|
||||
expect([TagName.count, Tag.count]).to eq(counts)
|
||||
end
|
||||
|
||||
it 'resolves aliases and keeps tag sections separate from canonical names' do
|
||||
canonical = create_tag!('虹夏', :character)
|
||||
TagName.create!(name: 'にじか', canonical: canonical.tag_name)
|
||||
create_tag!('動画', :meta)
|
||||
|
||||
plan = described_class.new(
|
||||
attributes: {
|
||||
url: 'https://example.com/video',
|
||||
title: 'video',
|
||||
tags: '動画 にじか[0:10-0:20]',
|
||||
duration: '1:00',
|
||||
parent_post_ids: '' }).build!
|
||||
|
||||
expect(plan[:tags].split).to include('動画', '虹夏[0:10-0:20]')
|
||||
expect(plan[:display_tags]).to include(
|
||||
{ name: '虹夏',
|
||||
category: 'character',
|
||||
section_literals: ['[0:10-0:20]'] })
|
||||
expect(plan[:video_ms]).to eq(60_000)
|
||||
end
|
||||
|
||||
it 'validates a new tag name without persisting it' do
|
||||
long_name = 'a' * 256
|
||||
counts = [TagName.count, Tag.count]
|
||||
|
||||
expect {
|
||||
described_class.new(
|
||||
attributes: {
|
||||
url: 'https://example.com/post',
|
||||
title: 'title',
|
||||
tags: long_name,
|
||||
parent_post_ids: '' }).build!
|
||||
}.to raise_error(ActiveRecord::RecordInvalid) { |error|
|
||||
expect(error.record.errors[:tags]).not_to be_empty
|
||||
}
|
||||
expect([TagName.count, Tag.count]).to eq(counts)
|
||||
end
|
||||
|
||||
it 'ignores duration when the planned tags do not include video' do
|
||||
plan = described_class.new(
|
||||
attributes: {
|
||||
url: 'https://example.com/post',
|
||||
title: 'title',
|
||||
tags: 'ordinary_tag',
|
||||
duration: 'invalid',
|
||||
parent_post_ids: '' }).build!
|
||||
|
||||
expect(plan[:duration]).to eq('invalid')
|
||||
expect(plan[:video_ms]).to be_nil
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,78 @@
|
||||
require 'rails_helper'
|
||||
require 'base64'
|
||||
|
||||
RSpec.describe PostCreator do
|
||||
let(:actor) { create(:user, :member) }
|
||||
|
||||
def real_thumbnail_upload
|
||||
gif = Base64.decode64('R0lGODdhAQABAIABAP///wAAACwAAAAAAQABAAACAkQBADs=')
|
||||
|
||||
Rack::Test::UploadedFile.new(
|
||||
StringIO.new(gif),
|
||||
'image/gif',
|
||||
original_filename: 'thumbnail.gif')
|
||||
end
|
||||
|
||||
before do
|
||||
allow(Tag).to receive(:normalise_tags!).and_return({ tags: [], sections: {} })
|
||||
allow(TagVersioning).to receive(:record_tag_snapshots!)
|
||||
allow(Tag).to receive(:expand_parent_tags).and_return([])
|
||||
allow(PostVersionRecorder).to receive(:record!)
|
||||
end
|
||||
|
||||
it 'prefers thumbnail_base over an explicit upload' do
|
||||
expect(Post).not_to receive(:resized_thumbnail_attachment)
|
||||
allow(Post).to receive(:remote_thumbnail_attachment).and_return(
|
||||
io: StringIO.new('remote'),
|
||||
filename: 'resized_thumbnail.jpg',
|
||||
content_type: 'image/jpeg')
|
||||
|
||||
post = described_class.new(
|
||||
actor:,
|
||||
attributes: {
|
||||
title: 'title',
|
||||
url: 'https://example.com/post',
|
||||
thumbnail: real_thumbnail_upload,
|
||||
thumbnail_base: 'https://example.com/thumb.jpg',
|
||||
tags: '' }).create!
|
||||
|
||||
expect(post.thumbnail).to be_attached
|
||||
expect(post.thumbnail_base).to eq('https://example.com/thumb.jpg')
|
||||
end
|
||||
|
||||
it 'uses the common remote thumbnail attach path when thumbnail_base is given' do
|
||||
expect(Post).to receive(:remote_thumbnail_attachment)
|
||||
.with('https://example.com/thumb.jpg')
|
||||
.and_return(
|
||||
io: StringIO.new('thumbnail'),
|
||||
filename: 'thumbnail.jpg',
|
||||
content_type: 'image/jpeg')
|
||||
|
||||
post = described_class.new(
|
||||
actor:,
|
||||
attributes: {
|
||||
title: 'title',
|
||||
url: 'https://example.com/post',
|
||||
thumbnail_base: 'https://example.com/thumb.jpg',
|
||||
tags: '' }).create!
|
||||
|
||||
expect(post.thumbnail_base).to eq('https://example.com/thumb.jpg')
|
||||
expect(post.thumbnail).to be_attached
|
||||
end
|
||||
|
||||
it 'does not create a post when remote thumbnail fetch fails' do
|
||||
allow(Post).to receive(:remote_thumbnail_attachment)
|
||||
.and_raise(Post::RemoteThumbnailFetchFailed, 'サムネール画像を取得できませんでした．')
|
||||
creator = described_class.new(
|
||||
actor:,
|
||||
attributes: {
|
||||
title: 'title',
|
||||
url: 'https://example.com/post',
|
||||
thumbnail_base: 'https://example.com/thumb.jpg',
|
||||
tags: '' })
|
||||
|
||||
post_count = Post.count
|
||||
expect { creator.create! }.to raise_error(Post::RemoteThumbnailFetchFailed)
|
||||
expect(Post.count).to eq(post_count)
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,187 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe PostImportPreviewer do
|
||||
before do
|
||||
allow(Preview::UrlSafety).to receive(:validate) do |url|
|
||||
[URI.parse(url), ['8.8.8.8']]
|
||||
end
|
||||
end
|
||||
|
||||
def row(source_row:, url:, attributes: { }, provenance: { }, **values)
|
||||
{
|
||||
source_row:,
|
||||
url:,
|
||||
attributes:,
|
||||
provenance:
|
||||
}.merge(values)
|
||||
end
|
||||
|
||||
describe '#preview_rows' do
|
||||
it 'marks an existing post for skipping without fetching metadata' do
|
||||
existing = create(:post, url: 'https://example.com/existing')
|
||||
expect(PostMetadataFetcher).not_to receive(:fetch)
|
||||
|
||||
result = described_class.new.preview_rows(rows: [
|
||||
row(source_row: 1, url: 'https://EXAMPLE.com/existing/')
|
||||
]).first
|
||||
|
||||
expect(result).to include(
|
||||
url: 'https://example.com/existing',
|
||||
skip_reason: 'existing',
|
||||
existing_post_id: existing.id,
|
||||
validation_errors: {}
|
||||
)
|
||||
expect(result.fetch(:field_warnings)).to eq({})
|
||||
end
|
||||
|
||||
it 'reports list duplicates as URL errors instead of existing skips' do
|
||||
create(:post, url: 'https://example.com/duplicate')
|
||||
expect(PostMetadataFetcher).not_to receive(:fetch)
|
||||
|
||||
results = described_class.new.preview_rows(rows: [
|
||||
row(source_row: 1, url: 'https://example.com/duplicate'),
|
||||
row(source_row: 2, url: 'https://EXAMPLE.com/duplicate/')
|
||||
])
|
||||
|
||||
expect(results.map { _1[:validation_errors] }).to all(
|
||||
include(url: ['URL が重複しています．'])
|
||||
)
|
||||
expect(results).to all(include(skip_reason: nil, existing_post_id: nil))
|
||||
end
|
||||
|
||||
it 'does not fetch metadata for an unsafe URL' do
|
||||
allow(Preview::UrlSafety).to receive(:validate)
|
||||
.and_raise(Preview::UrlSafety::UnsafeUrl, '安全でない接続先は使用できません．')
|
||||
expect(PostMetadataFetcher).not_to receive(:fetch)
|
||||
|
||||
result = described_class.new.preview_rows(rows: [
|
||||
row(source_row: 1, url: 'https://unsafe.example/post')
|
||||
]).first
|
||||
|
||||
expect(result.fetch(:validation_errors)).to include(
|
||||
url: ['安全でない接続先は使用できません．']
|
||||
)
|
||||
end
|
||||
|
||||
it 'applies metadata to automatic fields and recognises metadata tags' do
|
||||
Tag.create!(name: 'known-tag', category: :general)
|
||||
allow(PostMetadataFetcher).to receive(:fetch).and_return(
|
||||
title: 'metadata title',
|
||||
thumbnail_base: 'https://example.com/thumb.jpg',
|
||||
duration: '2',
|
||||
tags: 'known-tag'
|
||||
)
|
||||
|
||||
result = described_class.new.preview_rows(rows: [
|
||||
row(source_row: 1, url: 'https://example.com/new')
|
||||
]).first
|
||||
|
||||
expect(result.fetch(:attributes)).to include(
|
||||
'title' => 'metadata title',
|
||||
'thumbnail_base' => 'https://example.com/thumb.jpg',
|
||||
'duration' => '2',
|
||||
'tags' => 'known-tag'
|
||||
)
|
||||
expect(result.fetch(:field_warnings)).not_to have_key('tags')
|
||||
expect(PostMetadataFetcher).to have_received(:fetch).once
|
||||
end
|
||||
|
||||
it 'preserves manual empty values when metadata is available' do
|
||||
allow(PostMetadataFetcher).to receive(:fetch).and_return(
|
||||
title: 'metadata title',
|
||||
tags: 'metadata-tag'
|
||||
)
|
||||
|
||||
result = described_class.new.preview_rows(rows: [
|
||||
row(
|
||||
source_row: 1,
|
||||
url: 'https://example.com/manual',
|
||||
attributes: { title: '', tags: '' },
|
||||
provenance: { title: 'manual', tags: 'manual' },
|
||||
tag_sources: { automatic: 'old-tag', manual: '' }
|
||||
)
|
||||
]).first
|
||||
|
||||
expect(result.fetch(:attributes)).to include('title' => '', 'tags' => '')
|
||||
expect(result.fetch(:tag_sources)).to include('manual' => '')
|
||||
end
|
||||
|
||||
it 'clears warnings derived from an old metadata URL' do
|
||||
allow(PostMetadataFetcher).to receive(:fetch).and_return(
|
||||
title: 'new title',
|
||||
thumbnail_base: 'https://example.com/new-thumb.jpg',
|
||||
tags: ''
|
||||
)
|
||||
|
||||
result = described_class.new.preview_rows(rows: [
|
||||
row(
|
||||
source_row: 1,
|
||||
url: 'https://example.com/new-url',
|
||||
attributes: { title: 'old title', tags: 'old-tag' },
|
||||
provenance: { title: 'automatic', tags: 'automatic' },
|
||||
tag_sources: { automatic: 'old-tag', manual: '' },
|
||||
metadata_url: 'https://example.com/old-url',
|
||||
field_warnings: { tags: ['old tag warning'], url: ['old fetch warning'] },
|
||||
base_warnings: ['old base warning']
|
||||
)
|
||||
]).first
|
||||
|
||||
expect(result.fetch(:attributes)).to include('title' => 'new title', 'tags' => '')
|
||||
expect(result.fetch(:base_warnings)).to eq([])
|
||||
expect(result.fetch(:field_warnings).values.flatten)
|
||||
.not_to include('old tag warning', 'old fetch warning')
|
||||
end
|
||||
|
||||
it 'isolates an unexpected metadata failure to the affected URL' do
|
||||
allow(PostMetadataFetcher).to receive(:fetch) do |url|
|
||||
raise 'fetch failure' if url.include?('failure')
|
||||
|
||||
{ title: 'successful title', tags: '' }
|
||||
end
|
||||
allow(Rails.logger).to receive(:error)
|
||||
|
||||
results = described_class.new.preview_rows(rows: [
|
||||
row(source_row: 1, url: 'https://example.com/failure'),
|
||||
row(source_row: 2, url: 'https://example.com/success')
|
||||
])
|
||||
|
||||
expect(results[0].fetch(:field_warnings)).to include(
|
||||
'url' => [described_class::METADATA_FETCH_WARNING]
|
||||
)
|
||||
expect(results[1].fetch(:attributes)).to include('title' => 'successful title')
|
||||
end
|
||||
|
||||
it 'turns an unsafe URL detected during metadata fetching into a URL error' do
|
||||
allow(PostMetadataFetcher).to receive(:fetch)
|
||||
.and_raise(Preview::UrlSafety::UnsafeUrl, '安全でない接続先です．')
|
||||
allow(Rails.logger).to receive(:info)
|
||||
|
||||
result = described_class.new.preview_rows(rows: [
|
||||
row(source_row: 1, url: 'https://example.com/redirects-to-private')
|
||||
]).first
|
||||
|
||||
expect(result.fetch(:validation_errors)).to include(
|
||||
url: ['安全でない接続先です．']
|
||||
)
|
||||
end
|
||||
|
||||
it 'validates parent IDs from the preloaded set' do
|
||||
parent = create(:post)
|
||||
|
||||
results = described_class.new.preview_rows(
|
||||
rows: [
|
||||
row(source_row: 1, url: 'https://example.com/valid-parent',
|
||||
attributes: { parent_post_ids: parent.id.to_s }),
|
||||
row(source_row: 2, url: 'https://example.com/missing-parent',
|
||||
attributes: { parent_post_ids: '999999' })
|
||||
],
|
||||
fetch_metadata: false
|
||||
)
|
||||
|
||||
expect(results[0].fetch(:validation_errors)).not_to have_key(:parent_post_ids)
|
||||
expect(results[1].fetch(:validation_errors)).to include(
|
||||
parent_post_ids: ['存在しない親投稿 Id. があります．']
|
||||
)
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,123 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe PostMetadataFetcher do
|
||||
Response = Struct.new(:body)
|
||||
|
||||
def fetch_with_published_time(value, url: 'https://example.com/video')
|
||||
html = <<~HTML
|
||||
<html><head>
|
||||
<meta property="article:published_time" content="#{ value }">
|
||||
</head></html>
|
||||
HTML
|
||||
uri = URI.parse(url)
|
||||
allow(Preview::UrlSafety).to receive(:validate).with(url).and_return([uri, ['8.8.8.8']])
|
||||
allow(Preview::HttpFetcher).to receive(:fetch).and_return(Response.new(html))
|
||||
allow(Preview::HtmlMetadataExtractor).to receive(:extract)
|
||||
.and_return(title: 'title', image_url: nil)
|
||||
|
||||
described_class.fetch(url)
|
||||
end
|
||||
|
||||
it 'builds ranges explicitly for year, month, and day precision' do
|
||||
year = fetch_with_published_time('2024')
|
||||
month = fetch_with_published_time('2024-02')
|
||||
day = fetch_with_published_time('2024-02-03')
|
||||
|
||||
expect(year).to include(
|
||||
original_created_from: Time.zone.local(2024, 1, 1).iso8601,
|
||||
original_created_before: Time.zone.local(2025, 1, 1).iso8601
|
||||
)
|
||||
expect(month).to include(
|
||||
original_created_from: Time.zone.local(2024, 2, 1).iso8601,
|
||||
original_created_before: Time.zone.local(2024, 3, 1).iso8601
|
||||
)
|
||||
expect(day).to include(
|
||||
original_created_from: Time.zone.local(2024, 2, 3).iso8601,
|
||||
original_created_before: Time.zone.local(2024, 2, 4).iso8601
|
||||
)
|
||||
end
|
||||
|
||||
it 'rounds second and fractional-second timestamps down to one-minute ranges' do
|
||||
result = fetch_with_published_time('2024-02-03T12:34:56.123+02:30')
|
||||
|
||||
expect(Time.iso8601(result.fetch(:original_created_from)))
|
||||
.to eq(Time.iso8601('2024-02-03T12:34:00+02:30'))
|
||||
expect(
|
||||
Time.iso8601(result.fetch(:original_created_before)) -
|
||||
Time.iso8601(result.fetch(:original_created_from))
|
||||
).to eq(60)
|
||||
expect(result.fetch(:original_created_from)).not_to include('.')
|
||||
end
|
||||
|
||||
it 'treats hour-precision timestamps as one-hour ranges with and without offsets' do
|
||||
offset_result = fetch_with_published_time('2024-02-03T12+02:30')
|
||||
local_result = fetch_with_published_time('2024-02-03T12')
|
||||
|
||||
expect(Time.iso8601(offset_result.fetch(:original_created_from)))
|
||||
.to eq(Time.iso8601('2024-02-03T12:00:00+02:30'))
|
||||
expect(
|
||||
Time.iso8601(offset_result.fetch(:original_created_before)) -
|
||||
Time.iso8601(offset_result.fetch(:original_created_from))
|
||||
).to eq(3600)
|
||||
expect(local_result.fetch(:original_created_from)).to eq('2024-02-03T12:00:00Z')
|
||||
expect(local_result.fetch(:original_created_before)).to eq('2024-02-03T13:00:00Z')
|
||||
end
|
||||
|
||||
it 'adds platform tags for known video URLs' do
|
||||
result = fetch_with_published_time('2024', url: 'https://youtu.be/abc123')
|
||||
|
||||
expect(result.fetch(:tags)).to eq('動画 YouTube')
|
||||
end
|
||||
|
||||
it 'returns nil dates for an invalid timestamp' do
|
||||
result = fetch_with_published_time('not-a-time')
|
||||
|
||||
expect(result.fetch(:original_created_from)).to be_nil
|
||||
expect(result.fetch(:original_created_before)).to be_nil
|
||||
end
|
||||
|
||||
it 'returns nil dates for an invalid timestamp second value' do
|
||||
result = fetch_with_published_time('2024-02-03T12:34:99Z')
|
||||
|
||||
expect(result.fetch(:original_created_from)).to be_nil
|
||||
expect(result.fetch(:original_created_before)).to be_nil
|
||||
end
|
||||
|
||||
it 'returns nil dates for invalid calendar dates and invalid hour values' do
|
||||
invalid_day = fetch_with_published_time('2024-02-31T12:00Z')
|
||||
invalid_leap = fetch_with_published_time('2023-02-29T12:00Z')
|
||||
invalid_hour = fetch_with_published_time('2024-02-29T24:00Z')
|
||||
|
||||
[invalid_day, invalid_leap, invalid_hour].each do |result|
|
||||
expect(result.fetch(:original_created_from)).to be_nil
|
||||
expect(result.fetch(:original_created_before)).to be_nil
|
||||
end
|
||||
end
|
||||
|
||||
it 'accepts a valid leap-day timestamp at minute precision' do
|
||||
result = fetch_with_published_time('2024-02-29T12:34Z')
|
||||
|
||||
expect(result.fetch(:original_created_from)).to eq('2024-02-29T12:34:00Z')
|
||||
expect(result.fetch(:original_created_before)).to eq('2024-02-29T12:35:00Z')
|
||||
end
|
||||
|
||||
it 'serialises metadata duration as the same seconds string contract used by forms' do
|
||||
html = <<~HTML
|
||||
<html><head>
|
||||
<meta property="article:published_time" content="2024-02-03T12:34:56.123+02:30">
|
||||
<meta property="og:video:duration" content="2.5">
|
||||
</head></html>
|
||||
HTML
|
||||
uri = URI.parse('https://example.com/video')
|
||||
allow(Preview::UrlSafety).to receive(:validate)
|
||||
.with('https://example.com/video')
|
||||
.and_return([uri, ['8.8.8.8']])
|
||||
allow(Preview::HttpFetcher).to receive(:fetch).and_return(Response.new(html))
|
||||
allow(Preview::HtmlMetadataExtractor).to receive(:extract)
|
||||
.and_return(title: 'title', image_url: nil)
|
||||
|
||||
result = described_class.fetch('https://example.com/video')
|
||||
|
||||
expect(result.fetch(:duration)).to eq('2.5')
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,52 @@
|
||||
require 'rails_helper'
|
||||
require 'base64'
|
||||
require 'tempfile'
|
||||
|
||||
RSpec.describe PostThumbnailUploadValidator do
|
||||
def with_upload bytes, content_type:, filename:
|
||||
tempfile = Tempfile.new(['thumbnail-upload', File.extname(filename)])
|
||||
tempfile.binmode
|
||||
tempfile.write(bytes)
|
||||
tempfile.rewind
|
||||
upload = ActionDispatch::Http::UploadedFile.new(
|
||||
tempfile:,
|
||||
filename:,
|
||||
type: content_type)
|
||||
yield upload
|
||||
ensure
|
||||
tempfile&.close!
|
||||
end
|
||||
|
||||
it 'accepts a raster upload after decoding and rewinds it' do
|
||||
gif = Base64.decode64('R0lGODdhAQABAIABAP///wAAACwAAAAAAQABAAACAkQBADs=')
|
||||
|
||||
with_upload(gif, content_type: 'image/gif', filename: 'thumbnail.gif') do |upload|
|
||||
expect { described_class.validate!(upload) }.not_to raise_error
|
||||
expect(upload.read(6)).to eq('GIF87a')
|
||||
end
|
||||
end
|
||||
|
||||
it 'rejects SVG content disguised as a raster MIME type' do
|
||||
with_upload(
|
||||
'<svg width="10" height="10"></svg>',
|
||||
content_type: 'image/png',
|
||||
filename: 'thumbnail.png') do |upload|
|
||||
expect { described_class.validate!(upload) }
|
||||
.to raise_error(
|
||||
described_class::InvalidUpload,
|
||||
'サムネイル画像の形式が不正です．')
|
||||
end
|
||||
end
|
||||
|
||||
it 'rejects non-raster content disguised as an image' do
|
||||
with_upload(
|
||||
'%PDF-1.7',
|
||||
content_type: 'image/png',
|
||||
filename: 'thumbnail.png') do |upload|
|
||||
expect { described_class.validate!(upload) }
|
||||
.to raise_error(
|
||||
described_class::InvalidUpload,
|
||||
'サムネイル画像の形式が不正です．')
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,27 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe PostUrlNormaliser do
|
||||
before do
|
||||
PostUrlSanitisationRule.unscoped.delete_all
|
||||
end
|
||||
|
||||
describe '.normalise' do
|
||||
it 'normalises the host and trailing slash before applying sanitisation rules' do
|
||||
PostUrlSanitisationRule.create!(
|
||||
priority: 10,
|
||||
source_pattern: '\\Ahttps://example\\.com/source\\z',
|
||||
replacement: 'https://example.com/canonical'
|
||||
)
|
||||
|
||||
result = described_class.normalise(' https://EXAMPLE.com/source/ ')
|
||||
|
||||
expect(result).to eq('https://example.com/canonical')
|
||||
end
|
||||
|
||||
it 'returns nil for unsupported or malformed URLs' do
|
||||
expect(described_class.normalise('ftp://example.com/file')).to be_nil
|
||||
expect(described_class.normalise('https://[')).to be_nil
|
||||
expect(described_class.normalise('https:/path')).to be_nil
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -2,7 +2,7 @@ require 'rails_helper'
|
||||
|
||||
RSpec.describe Preview::ThumbnailFetcher do
|
||||
describe '.fetch' do
|
||||
it 'rejects svg thumbnails' do
|
||||
it 'accepts svg thumbnails for the common safe rasterisation path' do
|
||||
page = Preview::HttpFetcher::Response.new(
|
||||
'<meta property="og:image" content="https://example.com/thumb.svg">',
|
||||
'text/html',
|
||||
@@ -12,8 +12,9 @@ RSpec.describe Preview::ThumbnailFetcher do
|
||||
'image/svg+xml',
|
||||
'https://example.com/thumb.svg')
|
||||
|
||||
allow(Preview::UrlSafety).to receive(:validate)
|
||||
.and_return([URI.parse('https://example.com/page'), ['203.0.113.10']])
|
||||
allow(Preview::UrlSafety).to receive(:validate) do |url|
|
||||
[URI.parse(url), ['203.0.113.10']]
|
||||
end
|
||||
allow(Preview::HttpFetcher).to receive(:fetch)
|
||||
.with('https://example.com/page', max_bytes: described_class::HTML_MAX_BYTES)
|
||||
.and_return(page)
|
||||
@@ -21,9 +22,7 @@ RSpec.describe Preview::ThumbnailFetcher do
|
||||
.with('https://example.com/thumb.svg')
|
||||
.and_return(svg)
|
||||
|
||||
expect {
|
||||
described_class.fetch('https://example.com/page')
|
||||
}.to raise_error(Preview::ThumbnailFetcher::GenerationFailed)
|
||||
expect(described_class.fetch('https://example.com/page')).to eq('<svg></svg>')
|
||||
end
|
||||
|
||||
it 'accepts allowed image content type with parameters' do
|
||||
@@ -32,12 +31,13 @@ RSpec.describe Preview::ThumbnailFetcher do
|
||||
'text/html',
|
||||
'https://example.com/page')
|
||||
image = Preview::HttpFetcher::Response.new(
|
||||
'jpeg-bytes',
|
||||
"\xFF\xD8\xFFjpeg-bytes".b,
|
||||
'image/jpeg; charset=binary',
|
||||
'https://example.com/thumb.jpg')
|
||||
|
||||
allow(Preview::UrlSafety).to receive(:validate)
|
||||
.and_return([URI.parse('https://example.com/page'), ['203.0.113.10']])
|
||||
allow(Preview::UrlSafety).to receive(:validate) do |url|
|
||||
[URI.parse(url), ['203.0.113.10']]
|
||||
end
|
||||
allow(Preview::HttpFetcher).to receive(:fetch)
|
||||
.with('https://example.com/page', max_bytes: described_class::HTML_MAX_BYTES)
|
||||
.and_return(page)
|
||||
@@ -45,7 +45,60 @@ RSpec.describe Preview::ThumbnailFetcher do
|
||||
.with('https://example.com/thumb.jpg')
|
||||
.and_return(image)
|
||||
|
||||
expect(described_class.fetch('https://example.com/page')).to eq('jpeg-bytes')
|
||||
expect(described_class.fetch('https://example.com/page'))
|
||||
.to eq("\xFF\xD8\xFFjpeg-bytes".b)
|
||||
end
|
||||
end
|
||||
|
||||
describe '.fetch_image_response' do
|
||||
it 'rejects an unsafe input URL before HTTP fetch' do
|
||||
allow(Preview::UrlSafety).to receive(:validate)
|
||||
.and_raise(Preview::UrlSafety::UnsafeUrl, '安全でない接続先は使用できません．')
|
||||
expect(Preview::HttpFetcher).not_to receive(:fetch)
|
||||
|
||||
expect {
|
||||
described_class.fetch_image_response('https://unsafe.example.com/thumb.jpg')
|
||||
}.to raise_error(Preview::UrlSafety::UnsafeUrl, '安全でない接続先は使用できません．')
|
||||
end
|
||||
|
||||
it 'rejects an unsafe redirect target' do
|
||||
allow(Preview::UrlSafety).to receive(:validate)
|
||||
.and_return([URI.parse('https://example.com/thumb.jpg'), ['8.8.8.8']])
|
||||
allow(Preview::HttpFetcher).to receive(:fetch)
|
||||
.with('https://example.com/thumb.jpg')
|
||||
.and_raise(Preview::UrlSafety::UnsafeUrl, '安全でない接続先は使用できません．')
|
||||
|
||||
expect {
|
||||
described_class.fetch_image_response('https://example.com/thumb.jpg')
|
||||
}.to raise_error(Preview::UrlSafety::UnsafeUrl, '安全でない接続先は使用できません．')
|
||||
end
|
||||
|
||||
it 'rejects an oversized image response' do
|
||||
allow(Preview::UrlSafety).to receive(:validate)
|
||||
.and_return([URI.parse('https://example.com/thumb.jpg'), ['8.8.8.8']])
|
||||
allow(Preview::HttpFetcher).to receive(:fetch)
|
||||
.with('https://example.com/thumb.jpg')
|
||||
.and_raise(Preview::HttpFetcher::ResponseTooLarge, 'too large')
|
||||
|
||||
expect {
|
||||
described_class.fetch_image_response('https://example.com/thumb.jpg')
|
||||
}.to raise_error(Preview::HttpFetcher::ResponseTooLarge)
|
||||
end
|
||||
|
||||
it 'rejects a non-image content type' do
|
||||
allow(Preview::UrlSafety).to receive(:validate)
|
||||
.and_return([URI.parse('https://example.com/thumb.jpg'), ['8.8.8.8']])
|
||||
allow(Preview::HttpFetcher).to receive(:fetch)
|
||||
.with('https://example.com/thumb.jpg')
|
||||
.and_return(
|
||||
Preview::HttpFetcher::Response.new(
|
||||
'<html></html>',
|
||||
'text/html',
|
||||
'https://example.com/thumb.jpg'))
|
||||
|
||||
expect {
|
||||
described_class.fetch_image_response('https://example.com/thumb.jpg')
|
||||
}.to raise_error(Preview::ThumbnailFetcher::GenerationFailed)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
@@ -10,6 +10,37 @@ RSpec.describe Youtube::Sync do
|
||||
allow(sync).to receive(:attach_thumbnail_if_needed!)
|
||||
end
|
||||
|
||||
describe '#attach_thumbnail_if_needed!' do
|
||||
it 'uses the common remote thumbnail attach path' do
|
||||
post = create(:post, thumbnail_base: nil)
|
||||
allow(sync).to receive(:attach_thumbnail_if_needed!).and_call_original
|
||||
|
||||
expect(post).to receive(:attach_thumbnail_from_url!)
|
||||
.with('https://example.com/thumb.jpg')
|
||||
|
||||
sync.send(
|
||||
:attach_thumbnail_if_needed!,
|
||||
post,
|
||||
'https://example.com/thumb.jpg')
|
||||
end
|
||||
|
||||
it 'retries on a later sync when the previous remote attach failed' do
|
||||
post = create(:post, thumbnail_base: nil)
|
||||
allow(sync).to receive(:attach_thumbnail_if_needed!).and_call_original
|
||||
expect(post).to receive(:attach_thumbnail_from_url!)
|
||||
.with('https://example.com/thumb.jpg')
|
||||
.twice
|
||||
.and_raise(Post::RemoteThumbnailFetchFailed, 'failed')
|
||||
|
||||
2.times do
|
||||
sync.send(
|
||||
:attach_thumbnail_if_needed!,
|
||||
post,
|
||||
'https://example.com/thumb.jpg')
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
describe '#sync!' do
|
||||
it 'returns without fetching video details when no video ids are discovered' do
|
||||
allow(sync).to receive(:query_terms).and_return([])
|
||||
@@ -76,7 +107,7 @@ RSpec.describe Youtube::Sync do
|
||||
sync.sync!
|
||||
end
|
||||
|
||||
it 'creates a YouTube post with default tags and no_deerjikist when no deerjikist mapping exists' do
|
||||
it 'creates a YouTube post with default tags when no deerjikist mapping exists' do
|
||||
Tag.tagme
|
||||
Tag.bot
|
||||
Tag.youtube
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
require "rails_helper"
|
||||
require 'rails_helper'
|
||||
|
||||
|
||||
RSpec.describe "nico:sync" do
|
||||
RSpec.describe 'nico:sync' do
|
||||
def stub_python(json_array)
|
||||
status = instance_double(Process::Status, success?: true)
|
||||
allow(Open3).to receive(:capture3).and_return([json_array.to_json, "", status])
|
||||
allow(Open3).to receive(:capture3).and_return([json_array.to_json, '', status])
|
||||
end
|
||||
|
||||
def create_tag!(name, category:)
|
||||
@@ -16,17 +16,21 @@ RSpec.describe "nico:sync" do
|
||||
NicoTagRelation.create!(nico_tag_id: nico_tag.id, tag_id: tag.id)
|
||||
end
|
||||
|
||||
it "既存 post を見つけて、nico tag と linked tag を追加し、差分が出たら bot を付ける" do
|
||||
it '既存 post を見つけて、nico tag と linked tag を追加し、差分が出たら bot を付ける' do
|
||||
# 既存 post（正規表現で拾われるURL）
|
||||
post = Post.create!(title: "old", url: "https://www.nicovideo.jp/watch/sm9", uploaded_user: nil)
|
||||
post = Post.create!(
|
||||
title: 'old',
|
||||
url: 'https://www.nicovideo.jp/watch/sm9',
|
||||
uploaded_user: nil
|
||||
)
|
||||
|
||||
# 既存の非nicoタグ（kept_non_nico_ids）
|
||||
kept_general = create_tag!("spec_kept", category: "general")
|
||||
kept_general = create_tag!('spec_kept', category: 'general')
|
||||
PostTag.create!(post: post, tag: kept_general)
|
||||
|
||||
# 追加される linked tag を準備（nico tag に紐付く一般タグ）
|
||||
linked = create_tag!("spec_linked", category: "general")
|
||||
nico = create_tag!("nico:AAA", category: "nico")
|
||||
linked = create_tag!('spec_linked', category: 'general')
|
||||
nico = create_tag!('nico:AAA', category: 'nico')
|
||||
link_nico_to_tag!(nico, linked)
|
||||
|
||||
# bot / tagme は task 内で使うので作っておく（Tag.bot/tagme がある前提）
|
||||
@@ -42,43 +46,91 @@ RSpec.describe "nico:sync" do
|
||||
'deleted_at' => '2026-01-31 00:00:00' }])
|
||||
|
||||
# 外部HTTPは今回「既存 post なので呼ばれない」はずだが、念のため塞ぐ
|
||||
allow(URI).to receive(:open).and_return(StringIO.new("<html></html>"))
|
||||
allow(URI).to receive(:open).and_return(StringIO.new('<html></html>'))
|
||||
|
||||
run_rake_task("nico:sync")
|
||||
run_rake_task('nico:sync')
|
||||
|
||||
post.reload
|
||||
active_tag_names = post.tags.joins(:tag_name).pluck("tag_names.name")
|
||||
active_tag_names = post.tags.joins(:tag_name).pluck('tag_names.name')
|
||||
|
||||
expect(active_tag_names).to include("spec_kept")
|
||||
expect(active_tag_names).to include("nico:AAA")
|
||||
expect(active_tag_names).to include("spec_linked")
|
||||
expect(active_tag_names).to include('spec_kept')
|
||||
expect(active_tag_names).to include('nico:AAA')
|
||||
expect(active_tag_names).to include('spec_linked')
|
||||
|
||||
expect(post.original_created_from).to eq(Time.iso8601('2026-01-01T03:34:00Z'))
|
||||
expect(post.original_created_before).to eq(Time.iso8601('2026-01-01T03:35:00Z'))
|
||||
|
||||
# 差分が出るので bot が付く（kept_non_nico_ids != desired_non_nico_ids）
|
||||
expect(active_tag_names).to include("bot操作")
|
||||
expect(active_tag_names).to include('bot操作')
|
||||
end
|
||||
|
||||
it "既存 post にあった古い nico tag は active から外され、履歴として discard される" do
|
||||
post = Post.create!(title: "old", url: "https://www.nicovideo.jp/watch/sm9", uploaded_user: nil)
|
||||
it '既存 post のサムネール取得に共通 attach 経路を使ふ' do
|
||||
post = Post.create!(
|
||||
title: 'old',
|
||||
url: 'https://www.nicovideo.jp/watch/sm9',
|
||||
uploaded_user: nil)
|
||||
Tag.bot
|
||||
Tag.tagme
|
||||
|
||||
stub_python([{ 'code' => 'sm9', 'title' => 't', 'tags' => [] }])
|
||||
allow(URI).to receive(:open) do
|
||||
StringIO.new(
|
||||
'<meta name="thumbnail" content="https://example.com/thumb.jpg">')
|
||||
end
|
||||
expect_any_instance_of(Post).to receive(:attach_thumbnail_from_url!)
|
||||
.with('https://example.com/thumb.jpg')
|
||||
|
||||
run_rake_task('nico:sync')
|
||||
end
|
||||
|
||||
it 'サムネール取得失敗後も次回同期で再試行できる' do
|
||||
post = Post.create!(
|
||||
title: 'old',
|
||||
url: 'https://www.nicovideo.jp/watch/sm9',
|
||||
uploaded_user: nil)
|
||||
Tag.bot
|
||||
Tag.tagme
|
||||
|
||||
stub_python([{ 'code' => 'sm9', 'title' => 't', 'tags' => [] }])
|
||||
allow(URI).to receive(:open) do
|
||||
StringIO.new(
|
||||
'<meta name="thumbnail" content="https://example.com/thumb.jpg">')
|
||||
end
|
||||
calls = 0
|
||||
allow_any_instance_of(Post).to receive(:attach_thumbnail_from_url!) do
|
||||
calls += 1
|
||||
raise Post::RemoteThumbnailFetchFailed, 'failed'
|
||||
end
|
||||
|
||||
2.times do
|
||||
run_rake_task('nico:sync')
|
||||
end
|
||||
expect(calls).to eq(2)
|
||||
end
|
||||
|
||||
it '既存 post にあった古い nico tag は active から外され、履歴として discard される' do
|
||||
post = Post.create!(
|
||||
title: 'old',
|
||||
url: 'https://www.nicovideo.jp/watch/sm9',
|
||||
uploaded_user: nil
|
||||
)
|
||||
|
||||
# 旧nicoタグ（今回の同期結果に含まれない）
|
||||
old_nico = create_tag!("nico:OLD", category: "nico")
|
||||
old_nico = create_tag!('nico:OLD', category: 'nico')
|
||||
old_pt = PostTag.create!(post: post, tag: old_nico)
|
||||
expect(old_pt.discarded_at).to be_nil
|
||||
|
||||
# 今回は NEW のみ欲しい
|
||||
new_nico = create_tag!("nico:NEW", category: "nico")
|
||||
new_nico = create_tag!('nico:NEW', category: 'nico')
|
||||
|
||||
# bot/tagme 念のため
|
||||
Tag.bot
|
||||
Tag.tagme
|
||||
|
||||
stub_python([{ "code" => "sm9", "title" => "t", "tags" => ["NEW"] }])
|
||||
allow(URI).to receive(:open).and_return(StringIO.new("<html></html>"))
|
||||
stub_python([{ 'code' => 'sm9', 'title' => 't', 'tags' => ['NEW'] }])
|
||||
allow(URI).to receive(:open).and_return(StringIO.new('<html></html>'))
|
||||
|
||||
run_rake_task("nico:sync")
|
||||
run_rake_task('nico:sync')
|
||||
|
||||
# OLD は active から外れる（discarded_at が入る）
|
||||
old_pts = PostTag.where(post_id: post.id, tag_id: old_nico.id).order(:id).to_a
|
||||
@@ -86,9 +138,9 @@ RSpec.describe "nico:sync" do
|
||||
|
||||
# NEW は active にいる
|
||||
post.reload
|
||||
active_names = post.tags.joins(:tag_name).pluck("tag_names.name")
|
||||
expect(active_names).to include("nico:NEW")
|
||||
expect(active_names).not_to include("nico:OLD")
|
||||
active_names = post.tags.joins(:tag_name).pluck('tag_names.name')
|
||||
expect(active_names).to include('nico:NEW')
|
||||
expect(active_names).not_to include('nico:OLD')
|
||||
end
|
||||
|
||||
def snapshot_tags(post)
|
||||
|
||||
新しい課題から参照
ユーザをブロックする