Hey team! Looking at the open connector PRs (Slack, OneDrive, Dropbox, file upload), there's a lot of repeated BaseConnector plumbing across them. I drafted what a @archestra/connector-sdk package + scaffold could look like to cut that. Open to an RFC issue, or do you prefer keeping connectors in-tree?

Hi @user

Pretty cool idea, @user or @user are the authors of this integration. We need to check with them

hey 👋 I'm not sure we need an external package for this. Do you have specific examples of "repeated BaseConnector plumbing"? If there's particular repetition you notice I think extracting that into BaseConnector and/or shared utils would be fine 🙂

Hey @user tks!

Fair, in-tree first makes sense while the interface still moves.

Context on where I see this going: short term extract the repeated bits into BaseConnector + utils. Medium term, once the Connector / SyncBatch / checkpoint shapes stabilize, that same code lives behind a public archestra/connector-sdk + create-archestra-connector scaffold (Backstage / Airflow providers / ESLint plugins style).

In Long term that opens a connector marketplace, third parties ship zendesk / hubspot / freshdesk without forking core, and Archestra captures distribution + revenue share on paid ones. So I'd treat the in-tree extraction as step 0 of that path, not a separate thing.

what I hit reading dropbox + onedrive + slack #3962 was

how would that work? at the end of the day, the code for the available connectors need to be in the codebase/docker-images themselves. what concrete problem is this trying to solve that exists today?

do you have any examples of "repeated bits"?

let me get here

sorry, came to my pc was on mobile

repetition I see in-tree:

validateConfig is the same skeleton in dropbox / onedrive / notion / jira / github (parseX + null check + return). slack diverged with per-field errors, which is itself a sign the simple skeleton hit its ceiling.

for example

dropbox: https://github.com/archestra-ai/archestra/blob/main/platform/backend/src/knowledge-base/connectors/dropbox/dropbox-connector.ts#L56-L64

onedrive: https://github.com/archestra-ai/archestra/blob/main/platform/backend/src/knowledge-base/connectors/onedrive/onedrive-connector.ts#L70-L78

imho could be BaseConnector.validateConfigWithSchema(schema, label).

testConnection has the same try/catch + log + return shape, but error formatting already drifts. dropbox, onedrive, notion use extractErrorMessage. Jira and github do error instanceof Error ? .message : String(error) inline.

dropbox: https://github.com/archestra-ai/archestra/blob/main/platform/backend/src/knowledge-base/connectors/dropbox/dropbox-connector.ts#L66-L79

and I think jira: https://github.com/archestra-ai/archestra/blob/main/platform/backend/src/knowledge-base/connectors/jira/jira-connector.ts#L65-L95

we could have something like BaseConnector.runConnectionTest({ label, probe })

I didnt dig all of then

just some notes, came at this from a community product angle (lower the bar for outside contributors, faster connector PR throughput) and ended up zooming in on where the real friction is, which is the boilerplate every new connector reintroduces

Parking the SDK/marketplace bit, that was me jumping ahead. What I see as Concrete today (imho):

ValidateConfig is the same skeleton in dropbox / onedrive / notion / jira / github (parse + null check + return).

TestConnection has the same try/catch shape but error formatting already drifts (extractErrorMessage in dropbox/onedrive/notion vs inline instanceof Error in jira/github). Slack diverged into per-field errors, which is the skeleton hitting its ceiling.

Proposing a small PR: BaseConnector.validateConfigWithSchema(schema, label) + BaseConnector.runConnectionTest({ label, probe }), migrating dropbox/onedrive/notion as proof. If it holds across the next 2-3 connectors, we can revisit whether the surface deserves a stable package later. wdyt?

I think those sound like great cleanup suggestions!

yeah I mean we can still brainstorm the sdk/marketplace portion, totally ok 🙂 personally I'm just not sure I grok the vision/benefits (my first impression is it maybe is a bit of overengineering?) - but maybe I am missing something in your vision.

maintaining an npm package would be for external codebases to consume?

Thanks!

yeah, npm package for outside repos to consume. one pushback though: distribution and SDK are separate axes.

Hi, thanks for the proposal! Why do we need outside repos to consume this package?

backstage ships plugins bundled into the final app at build time and still maintains @backstage/* on npm. the package is just like an API contract + types + scaffold,, not a plugin loader. "connectors live in the docker image" doesn't kill the SDK case

hey! sorry, i was in the middle of writing hehehe

@user external consumption is the eventual shape, not a today-need, for sure

an internal workspace package already captures most of the value (typed contract, scaffold, semver discipline)

npm publish only mattering if you'd want community contributors shipping niche connectors without going through the core PR cycle

customers writing private connectors for internal systems they don't plan to upstream, or splitting archestra's own connectors across repos for independent release cadence.

for exemple: community contributors who'd ship niche connectors but don't want to clone and build the whole platform to do it

worth noting workspace-vs-npm is a false split. imho monorepo workspaces can publish to npm, same package serves in-tree connectors + external consumers, marginal cost is one CI step. backstage runs exactly that setup, plugins still get bundled into the final app at build time

so the path I'd actually propose: BaseConnector cleanup PR first, extract the workspace package as the shape stabilizes, npm publish flips on whenever external authoring becomes desirable. the publish step is reversible and cheap, the API surface is the part that (i think 🙂 ) needs care.

It's an interesting proposal, but I don't see any need in this. Anyone who wants to have a private connector could just maintain their own work. No separate packages needed, it's just an extra-complexity. If you see any potential improvements in BaseConnector structure as a part of existing codebase - we will be happy to review it.

got it, thanks. parking the SDK/external thread, if a contributor pattern actually shows up in the wild later it'd be backed by evidence rather than speculation, until then it's not worth the extra surface (sorry guys for the blogpost in here)

opening the BaseConnector cleanup PR (validateConfigWithSchema + runConnectionTest, migrating dropbox/onedrive/notion as proof). will tag y'all when it's up

PR up, https://github.com/archestra-ai/archestra/pull/4374. lmk if you want me to split it differently.