Commit Graph

680 Commits

Author SHA1 Message Date
Rafael Silva 6cb7b4dd04 stree 2025-05-29 18:54:09 -03:00
Rafael Silva 2d6f4eab0b rubocop 2025-05-29 18:53:14 -03:00
Rafael Silva 5500b1c2ff tests 2025-05-29 18:10:56 -03:00
Roman Rizzi a73e9e6724
Add proper support for array types in structured outputs 2025-05-29 18:06:20 -03:00
Rafael Silva 3a691332fe rubocop 2025-05-29 15:58:13 -03:00
Rafael Silva c1fc3c706c cleanup 2025-05-29 14:42:50 -03:00
Rafael Silva 2312a6fc43 post rebase fixes 2025-05-29 14:42:50 -03:00
Rafael Silva 0e817a71ac cleaning up 2025-05-29 14:42:50 -03:00
Rafael Silva 6ad2f7b83f Working deduplication 2025-05-29 14:42:50 -03:00
Rafael Silva ad94cb7509 Dedup concepts 2025-05-29 14:42:50 -03:00
Rafael Silva 5fe78aa1ff dots are not commas 2025-05-29 14:42:50 -03:00
Rafael Silva 7fdd67fe78 pass context 2025-05-29 14:42:50 -03:00
Rafael Silva 170c657672 small fixes 2025-05-29 14:42:50 -03:00
Rafael Silva 1fec8dc6be FEATURE: Extend inferred concepts to include posts
* Adds support for concepts to be inferred from and applied to posts
* Replaces daily task with one that handles both topics and posts
* Adds database migration for posts_inferred_concepts join table
* Updates PersonaContext to include inferred concepts
2025-05-29 14:42:50 -03:00
Rafael Silva 04c6e55371 FEATURE: add inferred concepts system
This commit adds a new inferred concepts system that:
- Creates a model for storing concept labels that can be applied to topics
- Provides AI personas for finding new concepts and matching existing ones
- Adds jobs for generating concepts from popular topics
- Includes a scheduled job that automatically processes engaging topics
2025-05-29 14:42:48 -03:00
Natalie Tay 373e2305d6
FEATURE: Automatic translation and localization of posts, topics, categories (#1376)
Related: https://github.com/discourse/discourse-translator/pull/310

This commit includes all the jobs and event hooks to localize posts, topics, and categories.

A few notes:
- `feature_name: "translation"` because the site setting is `ai-translation` and module is `Translation`
- we will switch to proper ai-feature in the near future, and can consider using the persona_user as `localization.localizer_user_id`
- keeping things flat within the module for now as we will be moving to ai-feature soon and have to rearrange
- Settings renamed/introduced are:
  - ai_translation_backfill_rate (0)
  - ai_translation_backfill_limit_to_public_content (true)
  - ai_translation_backfill_max_age_days (5)
  - ai_translation_verbose_logs (false)
2025-05-29 17:28:06 +08:00
Roman Rizzi 01e29ca5d8
FIX: Bump persona's examples length (#1377) 2025-05-28 14:01:44 -03:00
Keegan George 297c64c3b8
DEV: Unhide spam detection setting (#1374)
## 🔍 Overview
We want to unhide `ai_spam_detection_enabled` setting so that we can retain staff action log features. However, we also want to ensure users cannot enable spam detection without having `AiModerationSetting.spam` present in the database.

In this update we unhide the setting, but also add a validator to ensure the necessary configuration is in place before allowing the setting to be enabled.
2025-05-28 07:50:56 -07:00
Sam 70b0db2871
FIX: improve researcher tool - fix topic filters (#1368)
* Small fix, reasoning is now available on Claude 4 models

* fix invalid filters should raise, topic filter not working

* fix spec so we are consistent
2025-05-26 16:00:44 +10:00
Roman Rizzi 0ce17a122f
FIX: Correctly pass tool_choice when using Claude models. (#1364)
The `ClaudePrompt` object couldn't access the original prompt's tool_choice attribute, affecting both Anthropic and Bedrock.
2025-05-23 10:36:52 -03:00
Sam cf220c530c
FIX: Improve MessageBus efficiency and correctly stop streaming (#1362)
* FIX: Improve MessageBus efficiency and correctly stop streaming

This commit enhances the message bus implementation for AI helper streaming by:

- Adding client_id targeting for message bus publications to ensure only the requesting client receives streaming updates
- Limiting MessageBus backlog size (2) and age (60 seconds) to prevent Redis bloat
- Replacing clearTimeout with Ember's cancel method for proper runloop management, we were leaking a stop
- Adding tests for client-specific message delivery

These changes improve memory usage and make streaming more reliable by ensuring messages are properly directed to the requesting client.

* composer suggestion needed a fix as well.

* backlog size of 2 is risky here cause same channel name is reused between clients
2025-05-23 16:23:06 +10:00
Roman Rizzi d72ad84f8f
FIX: Retry parsing escaped inner JSON to handle control chars. (#1357)
The structured output JSON comes embedded inside the API response, which is also a JSON. Since we have to parse the response to process it, any control characters inside the structured output are unescaped into regular characters, leading to invalid JSON and breaking during parsing. This change adds a retry mechanism that escapes
the string again if parsing fails, preventing the parser from breaking on malformed input and working around this issue.

For example:

```
  original = '{ "a": "{\\"key\\":\\"value with \\n newline\\"}" }'
  JSON.parse(original) => { "a" => "{\"key\":\"value with \n newline\"}" }
  # At this point, the inner JSON string contains an actual newline.
```
2025-05-21 11:25:59 -03:00
Roman Rizzi e207eba1a4
FIX: Don't dig on nil when checking for the gemini schema (#1356) 2025-05-21 08:30:47 -03:00
Sam 7db2589cc4
DEV: prompt engineering to improve citations (#1351) 2025-05-20 13:01:35 +10:00
Roman Rizzi 2fb691cba8
FEATURE: Triage can hide posts after adding them to the review queue (#1348) 2025-05-20 08:19:00 +10:00
Sam de0625571b
FIX: add safe navigation to serializer include conditions (#1349)
In some rare cases a post can exist without a topic
2025-05-19 18:55:55 -03:00
Keegan George 9ee82fd8be
DEV: Temporarily suppress diff animation as we fix issues (#1341)
The diff animation introduced in https://github.com/discourse/discourse-ai/pull/1332 and with attempts to improve it in https://github.com/discourse/discourse-ai/pull/1338 still has various issues. As we work on a fix, we want to revert the animation to simply stream the diff without animation so users are not left with a janky unusable experience.
2025-05-15 14:55:30 -07:00
Keegan George dfea784fc4
DEV: Improve diff streaming accuracy with safety checker (#1338)
This update adds a safety checker which scans the streamed updates. It ensures that incomplete segments of text are not sent yet over message bus as this will cause breakage with the diff streamer. It also updates the diff streamer to handle a thinking state for when we are waiting for message bus updates.
2025-05-15 11:38:46 -07:00
Roman Rizzi ff2e18f9ca
FIX: Structured output discrepancies. (#1340)
This change fixes two bugs and adds a safeguard.

The first issue is that the schema Gemini expected differed from the one sent, resulting in 400 errors when performing completions.

The second issue was that creating a new persona won't define a method
for `response_format`. This has to be explicitly defined when we wrap it inside the Persona class. Also, There was a mismatch between the default value and what we stored in the DB. Some parts of the code expected symbols as keys and others as strings.

Finally, we add a safeguard when, even if asked to, the model refuses to reply with a valid JSON. In this case, we are making a best-effort to recover and stream the raw response.
2025-05-15 11:32:10 -03:00
Sam 1b3fdad5c7
FEATURE: allow researcher to also research specific topics (#1339)
* FEATURE: allow researcher to also research specific topics

Also improve UI around research with more accurate info

* this ensures that under no conditions PMs will be included
2025-05-15 17:48:21 +10:00
Sam 2c6459429f
DEV: use a proper object for tool definition (#1337)
* DEV: use a proper object for tool definition

This moves away from using a loose hash to define tools, which
is error prone.

Instead given a proper object we will also be able to coerce the
return values to match tool definition correctly

* fix xml tools

* fix anthropic tools

* fix specs... a few more to go

* specs are passing

* FIX: coerce values for XML tool calls

* Update spec/lib/completions/tool_definition_spec.rb

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-05-15 17:32:39 +10:00
Sam c34fcc8a95
FEATURE: forum researcher persona for deep research (#1313)
This commit introduces a new Forum Researcher persona specialized in deep forum content analysis along with comprehensive improvements to our AI infrastructure.

Key additions:

    New Forum Researcher persona with advanced filtering and analysis capabilities
    Robust filtering system supporting tags, categories, dates, users, and keywords
    LLM formatter to efficiently process and chunk research results

Infrastructure improvements:

    Implemented CancelManager class to centrally manage AI completion cancellations
    Replaced callback-based cancellation with a more robust pattern
    Added systematic cancellation monitoring with callbacks

Other improvements:

    Added configurable default_enabled flag to control which personas are enabled by default
    Updated translation strings for the new researcher functionality
    Added comprehensive specs for the new components

    Renames Researcher -> Web Researcher

This change makes our AI platform more stable while adding powerful research capabilities that can analyze forum trends and surface relevant content.
2025-05-14 12:36:16 +10:00
Roman Rizzi aef84bc5bb
FEATURE: Examples support for personas. (#1334)
Examples simulate previous interactions with an LLM and come
right after the system prompt. This helps grounding the model and
producing better responses.
2025-05-13 10:06:16 -03:00
Rafael dos Santos Silva 67587367ab
FEATURE: New setting to control model for translations (#1333) 2025-05-12 12:12:30 -03:00
Sam cf45e6884c
FIX: persona triage should be logged to automation (#1326)
We were logging persona triage as "bot" in logs, causing some
confusions around real world usage

This amends it so we log usage to "automation - AUTOMATION NAME"
2025-05-08 12:51:36 +10:00
Sam 2a62658248
FEATURE: support configurable thinking tokens for Gemini (#1322) 2025-05-08 07:39:50 +10:00
Rafael dos Santos Silva 1b71330cea
FIX: Correct prompt format for img2text used in our AI Bot PDF Rag pipeline (#1323) 2025-05-07 16:48:56 -03:00
Roman Rizzi 5bc9fdc06b
FIX: Return structured output on non-streaming mode (#1318) 2025-05-06 15:34:30 -03:00
Keegan George adc2716cec
FIX: Invalid access error in logs (#1317) 2025-05-06 10:13:03 -07:00
Roman Rizzi c0a2d4c935
DEV: Use structured responses for summaries (#1252)
* DEV: Use structured responses for summaries

* Fix system specs

* Make response_format a first class citizen and update endpoints to support it

* Response format can be specified in the persona

* lint

* switch to jsonb and make column nullable

* Reify structured output chunks. Move JSON parsing to the depths of Completion

* Switch to JsonStreamingTracker for partial JSON parsing
2025-05-06 10:09:39 -03:00
Sam c6a307b473
FIX: handle unexpected errors when browsing web (#1314) 2025-05-06 18:12:26 +10:00
Sam e9fed4f5fa
FEATURE: ensure researcher and github helper know the date (#1312)
The date / time is important cause it creates a bias towards
action.

Without it, model may think it is getting future annoucements vs
up to date info.
2025-05-06 14:39:30 +10:00
Rafael dos Santos Silva 4eac377987
DEV: Zero delays on fake endpoint used in tests (#1311) 2025-05-05 17:47:32 -03:00
Roman Rizzi 48305dc7d3
FIX: resource_url replacemente in Persona's system prompt (#1310) 2025-05-05 11:41:04 -03:00
Sam a6fa619c31
FEATURE: enforce jpg/png for all images (#1309)
This ensures webp / gif are converted to png prior to sending
to LLMs given webp and gif are not evenly supported.

png/jpg is universally supported and are the only supported format.

longer term we need to add support for audio/video/pdf which is supported by some models.

* more specs
2025-05-05 17:46:37 +10:00
Sam 9196546f6f
FIX: better LLM feedback for image generation failures (#1306)
* FIX: handle error conditions when generating images gracefully

* FIX: also handle error for edit_image

* Update lib/inference/open_ai_image_generator.rb

Co-authored-by: Krzysztof Kotlarek <kotlarek.krzysztof@gmail.com>

* lint

---------

Co-authored-by: Krzysztof Kotlarek <kotlarek.krzysztof@gmail.com>
2025-05-01 19:25:38 +10:00
Sam 491dac298f
FIX: system persona state leaking between sites (#1304)
System personas leaned on reused classes, this was a problem
in a multisite environement cause state, such as "enabled"
ended up being reused between sites.

New implementation ensures state is pristine between sites in
a multisite

* more handling for new superclass story

* small oversight, display name should be used for display
2025-05-01 13:24:53 +10:00
Sam 7dc3c30fa4
FEATURE: correctly decorate AI bots (#1300)
AI bots come in 2 flavors

1. An LLM and LLM user, in this case we should decorate posts with persona name
2. A Persona user, in this case, in PMs we decorate with LLM name

(2) is a significant improvement, cause previously when creating a conversation
you could not tell which LLM you were talking to by simply looking at the post, you would
have to scroll to the top of the page.

* lint

* translation missing
2025-04-30 16:36:38 +10:00
Sam 09a6841480
FIX: s3 was missing a const (#1298) 2025-04-30 07:58:24 +10:00
Sam 17f04c76d8
FEATURE: add OpenAI image generation and editing capabilities (#1293)
This commit enhances the AI image generation functionality by adding support for:

1. OpenAI's GPT-based image generation model (gpt-image-1)
2. Image editing capabilities through the OpenAI API
3. A new "Designer" persona specialized in image generation and editing
4. Two new AI tools: CreateImage and EditImage

Technical changes include:
- Renaming `ai_openai_dall_e_3_url` to `ai_openai_image_generation_url` with a migration
- Adding `ai_openai_image_edit_url` setting for the image edit API endpoint
- Refactoring image generation code to handle both DALL-E and the newer GPT models
- Supporting multipart/form-data for image editing requests

* wild guess but maybe quantization is breaking the test sometimes

this increases distance

* Update lib/personas/designer.rb

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>

* simplify and de-flake code

* fix, in chat we need enough context so we know exactly what uploads a user uploaded.

* Update lib/personas/tools/edit_image.rb

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>

* cleanup downloaded files right away

* fix implementation

---------

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2025-04-29 17:38:54 +10:00