I Sandbox-Tested 1,800+ DeepSeek Harness Plugins. Here’s What Broke
The DeepSeek Harness plugin ecosystem is growing fast.
That sounds great until you actually try to install plugins at scale.
When I started building a searchable catalogue for DSH plugins, I assumed the hard part would be discovery:
- finding repositories
- reading metadata
- extracting install commands
- sorting by stars and categories
It wasn't.
The hard part was figuring out which information could actually be trusted.
After running sandbox installation checks on more than 1,800 plugins, I found several failure modes that were much more common than I expected.
That work eventually became DSH Marketplace:
But the interesting part is what broke along the way.
1. A package name in package.json does not mean the package exists
The first version of the pipeline was simple.
If a repository contained:
{
"name": "some-dsh-plugin"
}
then generating an installation command seemed reasonable:
dsh plugin --profile web add some-dsh-plugin
The problem is that repository metadata is not the npm registry.
Some packages simply did not exist.
Others were even worse.
A fork could inherit the upstream repository's package.json, including its package name.
Imagine:
author-a/plugin
↓ fork
author-b/plugin
The fork still contains:
{
"name": "author-a-plugin"
}
If a directory blindly trusts that metadata, the page for author-b/plugin may tell users to install:
dsh plugin --profile web add author-a-plugin
The command works.
But it installs someone else's package.
That is more dangerous than a broken command because it looks successful.
2. "Installation failed" does not mean "plugin is broken"
My first validation model was essentially:
success
failure
That quickly turned out to be useless.
Real installation results looked more like:
passed
needs-approval
not-a-layer
failed
timeout
rejected
Those distinctions matter.
needs-approval
Some packages contain build scripts that require explicit approval.
That does not necessarily mean the plugin is broken.
It may only need an allowBuilds entry before it can activate.
not-a-layer
A package may install successfully but not declare a DSH profile layer.
It could be:
- a theme
- a skill bundle
- a library
- an agent extension
Calling that "broken" would be misleading.
A useful validator needs to understand more than the process exit code.
3. Sometimes your validator is the thing that failed
This was the biggest lesson.
At one point, a large batch of plugins came back as failed.
The tempting conclusion was:
A huge percentage of DSH plugins are broken.
That conclusion was wrong.
After inspecting the logs, many failures came from the validation infrastructure itself:
- bad npm package claims
- npm registry throttling
- temporary network failures
- monorepo resolution
- changes in the probe during a long validation run
One batch produced more than a hundred failures that were mostly registry throttling.
If I had published those results directly, perfectly valid plugins would have been labeled as broken.
So the pipeline now tries to classify failures as:
OURS
or:
THEIRS
If the failure cannot be explained confidently, the negative verdict should not be published.
This became an important rule:
An unexplained failure is a validator problem until proven otherwise.
4. Testing third-party plugins means executing third-party code
There is another obvious problem with installation validation:
You are executing code you do not control.
npm packages can run lifecycle scripts such as:
{
"scripts": {
"postinstall": "..."
}
}
A malicious installation script can do much more than install dependencies.
So the validator cannot run in the same environment that holds production credentials.
The architecture now looks roughly like this:
Marketplace
↓
export plugins that need testing
↓
isolated validation job
↓
disposable containers
↓
installation results
↓
trusted job applies results
The job that runs third-party installation code holds no production database secrets.
The containers are also restricted:
- non-root
- no host mounts
- limited memory
- limited process count
- dropped capabilities
- disposable DSH profiles
The important part is not that the sandbox is "perfect".
It is that untrusted plugin code is separated from the credentials that matter.
5. Metadata needs continuous verification
Even after fixing the first batch, the problem does not disappear.
Repositories change.
Packages get unpublished.
Forks appear.
Install commands go stale.
So npm verification cannot be a one-time migration.
The catalogue now re-checks package claims as part of its recurring sync process.
The basic pipeline is:
discover
→ refresh metadata
→ verify npm claims
→ apply admission rules
→ sandbox install
→ classify failures
→ update results
→ deploy
This is more expensive than scraping a GitHub topic once.
But it produces much more useful data.
Why I ended up building DSH Marketplace
Originally, I wanted a simple plugin directory.
But a directory that only shows:
name
description
stars
GitHub URL
does not solve much.
GitHub already does that.
The useful questions are:
What does this plugin actually do?
Is the install command still valid?
Does the npm package belong to this repository?
Did the plugin fail, or did the validator fail?
Does installation require approval?
That is why DSH Marketplace gradually became more than a list of links.
Today it indexes 2,500+ DeepSeek Harness plugins, with 1,800+ sandbox installation checks.
You can browse it here:
The project is open source:
https://github.com/DshMarketPlace
It also works inside DSH
The Marketplace itself is available as a DSH plugin:
dsh plugin --profile web add dshmarketplace-plugin
There are also:
- a CLI
- Python SDK / CLI
- public API
- browser userscript for GitHub/npm
- the web marketplace
The public API does not require a key:
curl 'https://dshmarketplace.dev/api/v1/plugins?q=memory&limit=5'
The goal is to keep all of these surfaces backed by the same catalogue.
What I learned
The biggest lesson was not about Docker or npm.
It was about trust.
When building a software directory, it is easy to assume that metadata is truth.
But:
- README files become outdated
- package names can be inherited
- registries can disagree with repositories
- install commands can silently install the wrong thing
- automated tests can generate false negatives
At enough scale, every shortcut eventually becomes visible.
So the principle I ended up with is simple:
Trust metadata less. Verify behavior more.
DSH Marketplace is still early, but this validation pipeline has already been much more useful than simply collecting more repositories.
If you're using DeepSeek Harness, I'd be interested to know:
How do you currently decide whether a plugin is worth installing?
GitHub stars? Awesome lists? Community recommendations? Or do you test them yourself?
