Yes — if you want to restart the ML part from the beginning, this is actually a good point to simplify it properly. We do not need to redesign the whole project; we can rebuild the hybrid experiment cleanly around the idea you originally intended.

And yes, your understanding of PCA is correct:

Sentence-BERT gives a 384-dimensional embedding. PCA can be used to reduce those 384 dimensions to a smaller number of dimensions.

But 384 dimensions is not automatically "too high". With your dataset size, the question is whether keeping all 384 dimensions is useful and stable.

1. Start from the simplest possible architecture

I recommend starting with this:

                 CARBON CATALOGUE
                       │
              Fixed 80/20 split
                       │
          ┌────────────┴────────────┐
          │                         │
          ▼                         ▼
   STRUCTURED FEATURES          TEXT FEATURES
          │                         │
   Year, Weight, etc.       Product + Company
          │                   + Country
          │                         │
          │                         ▼
          │                  Sentence-BERT
          │                         │
          │                    384-D embedding
          │                         │
          └────────────┬────────────┘
                       ▼
                 FEATURE FUSION
                       │
                       ▼
             HYBRID FEATURE VECTOR
                       │
                       ▼
                  ML REGRESSOR
                       │
                       ▼
                  PCF prediction

That is the core hybrid model.

Don't introduce PCA yet.

First establish that this model works.

2. What exactly happens to the text?

Suppose one row contains:

Product Name:
Organic cotton T-shirt

Company:
Example Textile Ltd

Country:
Bangladesh

You combine those text fields:

Organic cotton T-shirt | Example Textile Ltd | Bangladesh

Then:

                  Combined text
                       │
                       ▼
               Sentence-BERT
                       │
                       ▼
          [e1, e2, e3, ..., e384]

So each row gets 384 numerical features.

For example, conceptually:

e1       = 0.034
e2       = -0.127
e3       = 0.082
...
e384     = 0.041

These are the embedding features.

3. Is 384-D too high?

Short answer: No.

384 dimensions is actually quite modest for a text embedding.

The bigger issue is your dataset size.

You have approximately:

866 total rows

and therefore roughly:

553 training
139 validation
174 test

So you have around 553 inner-training observations.

If you directly use:

384 embedding features

you have a relatively high feature-to-sample ratio.

It is not impossible, but it creates a reasonable research question:

Does dimensionality reduction improve generalisation?

That's exactly where PCA becomes useful.

4. PCA's role becomes very clear now

Think of PCA like this:

Sentence-BERT
     │
     ▼
384 dimensions
     │
     │ PCA
     ▼
smaller representation

For example:

384 → 200
384 → 100
384 → 75
384 → 50

PCA attempts to preserve as much of the variation/information in the original embedding as possible while representing it using fewer dimensions.

So PCA is not creating the hybrid model.

Instead:

Sentence-BERT creates the text representation. PCA optionally compresses that representation. Feature fusion creates the hybrid representation.

This distinction is very important.

5. Therefore, don't immediately test 50, 75, 100, 150, 200

This is where I would change the previous notebook.

If we start again, I recommend a two-step experiment.

Experiment 1 — Full embedding

Text
 ↓
Sentence-BERT
 ↓
384-D
 ↓
Fusion
 ↓
ML

This establishes your baseline hybrid model.

Experiment 2 — PCA

Then investigate whether PCA improves it:

Text
 ↓
Sentence-BERT
 ↓
384-D
 ↓
PCA
 ↓
Fusion
 ↓
ML

Now PCA has a clear scientific purpose.

6. What should PCA dimensions be?

Don't arbitrarily choose many values simply because they are available.

I would use a small, justified set such as:

384
200
100
50

where:

• 384 = no PCA
• 200 = moderate reduction
• 100 = substantial reduction
• 50 = strong reduction

Then compare validation performance.

You could also select the PCA dimension based on explained variance, which is arguably more principled.

For example:

PCA → retain 90% variance

or

PCA → retain 95% variance

This is actually preferable academically because you are not saying:

"I randomly decided to use 50 dimensions."

Instead:

"The number of principal components was selected according to the amount of variance retained."

However, because your dissertation has a practical time constraint, I would first run 384/200/100/50 and inspect the results. We don't need to over-engineer this.

7. The important leakage issue

This is very important when you rebuild it.

PCA must be fitted only on the training data.

Not:

Train + Validation + Test
        ↓
      PCA

That would leak information.

Instead:

Inner training embeddings
        │
        ▼
    FIT PCA
        │
        ├──────────────► transform training
        │
        └──────────────► transform validation

Then, after selecting the configuration, the final model is evaluated on the untouched test set.

So:

TRAIN
 ↓
fit Sentence-BERT? → no training required if using pre-trained frozen SBERT
fit PCA
fit ML model
 ↓
VALIDATION
 ↓
select PCA + model
 ↓
FREEZE
 ↓
TEST

This is the clean experimental protocol.

8. Why keep the 553 / 139 / 174 split?

I would keep it.

It is actually one of the stronger parts of the previous notebook.

866 total
│
├── 692 development = 80%
│   │
│   ├── 553 inner training
│   └── 139 validation
│
└── 174 final test = 20%

The roles are:

You should not use the 174 test rows to decide whether PCA=50 or PCA=384 is better.

9. Then compare structured vs hybrid

This is where your dissertation becomes much stronger.

Don't only report:

"Our hybrid model achieved X."

Instead ask:

Baseline model

Structured features
        ↓
Random Forest / XGBoost / etc.
        ↓
PCF

Hybrid model

Structured features
        +
Sentence-BERT text representation
        ↓
Feature fusion
        ↓
Same ML model
        ↓
PCF

Then:

             Baseline       Hybrid
RMSE            X             Y
MAE             X             Y
R²              X             Y

Now you can answer the real research question:

Does incorporating semantic information from textual PCF metadata improve PCF prediction compared with structured features alone?

That is much more academically meaningful than simply testing 25 combinations.

10. And the "best ML model" issue

I recommend we handle this in a controlled way.

For example:

                 STRUCTURED
                     │
          ┌──────────┼──────────┐
          ▼          ▼          ▼
        Ridge       RF       XGBoost
          │          │          │
          └──────────┼──────────┘
                     ▼
              Best baseline

Then:

             HYBRID FEATURES
                    │
          ┌─────────┼─────────┐
          ▼         ▼         ▼
        Ridge      RF      XGBoost
          │         │         │
          └─────────┼─────────┘
                    ▼
              Best hybrid

You can retain LightGBM and CatBoost if those are already part of your project methodology, but don't let the number of models obscure the central experiment.

11. So the complete experiment becomes

I would freeze this as your new ML workflow:

STEP 1
Carbon Catalogue
       │
       ▼
Fixed 80/20 split
       │
       ├─────────────────────┐
       │                     │
       ▼                     ▼
Structured branch       Text branch
       │                     │
       │              Sentence-BERT
       │                     │
       │                  384-D
       │                     │
       │                Optional PCA
       │                     │
       └──────────┬──────────┘
                  ▼
            Feature Fusion
                  │
                  ▼
          Hybrid Feature Set
                  │
                  ▼
            ML Regression
                  │
                  ▼
             Validation
                  │
                  ▼
          Select best setup
                  │
                  ▼
                Freeze
                  │
                  ▼
          Final 20% test
                  │
                  ▼
          Final performance

12. My recommendation about PCA

I would not decide now that 384 is too high.

Let the experiment answer that.

Run:

Hybrid-384
Hybrid-PCA-200
Hybrid-PCA-100
Hybrid-PCA-50

and compare validation performance.

If 384 performs best:

Keep 384.

If 100 performs similarly or better:

Use 100 and explain that dimensionality reduction achieved comparable/better predictive performance with a lower-dimensional representation.

If 50 performs best:

Use 50.

So PCA becomes an empirical dimensionality-reduction experiment, rather than an assumption.

13. One thing I would change from the previous notebook

I would make the notebook extremely explicit with four sections:

01. DATA SPLIT
02. STRUCTURED FEATURES
03. SENTENCE-BERT TEXT EMBEDDINGS
04. HYBRID FEATURE FUSION
05. PCA EXPERIMENT
06. MODEL COMPARISON
07. MODEL SELECTION
08. FINAL TEST

Then anybody opening the notebook can understand exactly what is happening.

The core idea in one sentence

Sentence-BERT converts textual PCF metadata into a 384-dimensional semantic representation; PCA is optionally used to reduce this representation; the resulting text features are concatenated with structured PCF features to form a hybrid feature set, which is then used by a regression model to predict PCF.

That is the model I recommend you build from the beginning.

And yes, we can restart the notebook from Step 1 and build it cleanly rather than trying to repair the current complicated version.