Explore having the LLM return content in JSON schema instead of WordPress block markup#1119
Explore having the LLM return content in JSON schema instead of WordPress block markup#1119dkotter wants to merge 11 commits into
Conversation
…n JSON. Modify our Providers to request JSON returned responses
✅ WordPress Plugin Check Report
📊 ReportAll checks passed! No errors or warnings found. 🤖 Generated by WordPress Plugin Check Action • Learn more about Plugin Check |
github-actions
Bot
added
the
needs:refresh
github-actions
Bot
added
needs:refresh
github-actions
Bot
removed
the
needs:refresh
peterwilsoncc
left a comment
peterwilsoncc
left a comment
There was a problem hiding this comment.
I've added some notes inline re validation and handling the old format in the event the filters are used.
The backcompat would need to be handled in the JS too, so if it's all a bit messy then maybe the filters can be renamed to include _json or similar so out of date plugins don't trigger errors and the use the engine instead.
| * @param mixed $value Decoded block tree, or a nested value within it. | ||
| * @return mixed Sanitized value. | ||
| */ | ||
| function sanitize_generated_block_tree( $value ) { |
There was a problem hiding this comment.
Is it possible to ensure that blocks with specific parent requirements are enforced, eg column is a child of columns? It's in the instructions but it would be good to remove the element of trust.
| 'temperature' => 0.9, | ||
| 'messages' => $messages, | ||
| 'temperature' => 0.9, | ||
| 'response_format' => array( 'type' => 'json_object' ), |
There was a problem hiding this comment.
Document introduction of response_format in X.X.X and that prior versions require block formatted code.
🔢 Applies to other providers too.
| // preserved when re-encoded rather than becoming "props":[]. | ||
| $decoded = json_decode( $content ); | ||
| if ( null === $decoded || JSON_ERROR_NONE !== json_last_error() ) { | ||
| return new WP_Error( 'invalid_content_response', esc_html__( 'The generated content was not in the expected format. Please try again.', 'classifai' ) ); |
There was a problem hiding this comment.
For developers using the filter and returning block formatted code, a check here for <!-- wp: and gracefully falling back to the older format would be lovely.
🔢 Applies to other providers too.
github-actions
Bot
added
the
needs:refresh
|
@dkotter thanks for the PR! Could you please rebase your PR on top of the latest changes in the base branch? |
2 participants
Description of the Change
For the Content Generation Feature, we specify in our prompt that the content returned should follow the WordPress block markup (with some examples). This seems to work but is fairly fragile as it relies on the LLM knowing what this block markup should look like.
There's a few alternatives to that approach and this PR is exploring one of those: having the LLM return content formatted as JSON and then having us parse that into valid block markup. In theory this should be more reliable and does open up a path where we could support a wider-range of block types (for example, blocks specific to a site or that come from a plugin that some sites may be using).
Some of the work here comes from the
@10up/block-renderer-corepackage (I was wanting to use the full suite but ran into issues getting that working in the block editor cc / @fabiankaegy). Our prompt is still static so won't include custom blocks but we now ask for things formatted as JSON and then parse that on the client-side to create our blocks.How to test the Change
npm i && npm run buildChangelog Entry
Credits
Props @fabiankaegy, @dkotter
Checklist: