The-OpenROAD-Project / The-OpenROAD-Project/OpenROAD
TritonCTS: macro insertion-delay balancing skews launch-dominated macros the wrong way; propose timing-driven skew scheduling
@arthurjolo is already working on this.
Since Jul 16, 2026.
- Dominant language
- Verilog
- Stars
- 3.1k
- Forks
- 1k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 136
Description
Description
TritonCTS::balanceMacroRegisterLatencies / LatencyBalancer pulls every abstracted-macro clock pin earlier by its Liberty clock_tree_path insertion delay. That is correct for a capture-dominated macro (core register → macro): delivering its clock early aligns its internal capture with the core registers.
It is backwards for a launch-dominated macro — a dual-port SRAM read (R0_clk) launching data into a core register. Pulling that clock earlier makes read data launch early, so it races into the downstream register: a boundary hold violation. A single fixed per-pin offset cannot satisfy a macro that both launches and captures, and neither Liberty corner works:
- max corner (today): worst boundary hold −221.9 ps on our design (ASAP7, OpenROAD
7ffbedfdbba). - min corner: −202 → −155 ps. Still deeply violated — the launch-side SRAMs need the clock later, which no "pull earlier" rule can produce.
This pathology is the dominant driver of our grt repair_timing hold phase: 608 hold-violated endpoints, 1456 hold buffers, ~3.2 h of a ~3.4 h combined repair. (The setup half of that grind is #10900; the two are orthogonal and compose.)
Suggested Solution
Choose each macro's clock arrival from measured slack instead of a fixed Liberty offset. This is Fishburn clock-skew scheduling; a hold-only, add-only specialization is enough to remove the pathology: after the base balance, push each macro's clock later just enough to clear its worst launch-hold violation, capped by its capture-hold room so capture hold never goes negative. Setup is deliberately ignored — at CTS it is premature; repair_timing owns it.
Measured (same design, iterated to convergence):
| base (today) | + timing-driven skew | |
|---|---|---|
| worst macro-boundary hold at CTS | −202 ps | −57.7 ps |
| hold-violated endpoints at CTS | 3590 | 79 |
grt repair_timing (setup+hold) |
~3.4 h | ~1.0 h |
| hold buffers inserted | 1456 | 404 |
The residual is ordinary closure of shallow reg→reg holds, not the macro-boundary avalanche it replaced. The setup phase is untouched, as intended.
Implementation notes (learned the hard way):
- Enumerate macros from the clock's sinks (
forEachSink), not the balancer's aggregate graph — a clustered macro tree folds sinks into leaf-builder aggregates that are invisible to the graph walk. - Gather launch-hold filter-free: iterate hold-violated
endpoints()+vertexWorstSlackPathand bin each to its launching macro (the RepairHold pattern).findPathEndswith anExceptionFromfilter leaves filtered-arrival tags that the subsequent buffer insertion invalidates; a later search dereferences stale tags and crashes. - Shift all clock pins of a dual-port macro (read + write) by the same delta.
- Realized with the existing delay-buffer mechanism — no DME rewrite. Flag-gated; the pure target math is a dependency-free unit pinned by a sub-1 s gtest.
Patch below (against 7ffbedfdbba, src/cts). Happy to turn it into a PR — and to hear how this fits with any useful-skew work already in progress; glad to test that against our design instead.
openroad-macro-clock-skew.patch
diff --git a/src/cts/BUILD b/src/cts/BUILD
index 90dcb02609..6e605fac1a 100644
--- a/src/cts/BUILD
+++ b/src/cts/BUILD
@@ -35,6 +35,13 @@ cc_library(
],
)
+cc_library(
+ name = "macro_latency_balance",
+ srcs = ["src/MacroLatencyBalance.cc"],
+ hdrs = ["src/MacroLatencyBalance.h"],
+ includes = ["src"],
+)
+
cc_library(
name = "cts",
srcs = [
@@ -59,6 +66,7 @@ cc_library(
"src",
],
deps = [
+ ":macro_latency_balance",
":private_hdrs",
"//src/dbSta",
"//src/dbSta:dbNetwork",
diff --git a/src/cts/src/CtsOptions.h b/src/cts/src/CtsOptions.h
index ef65632e08..9ca51beb0b 100644
--- a/src/cts/src/CtsOptions.h
+++ b/src/cts/src/CtsOptions.h
@@ -275,6 +275,8 @@ class CtsOptions : public odb::dbBlockCallBackObj
bool getObstructionAware() const { return obsAware_; }
void enableInsertionDelay(bool insDelay) { insertionDelay_ = insDelay; }
bool insertionDelayEnabled() const { return insertionDelay_; }
+ void setMacroSkew(bool enable) { macroSkew_ = enable; }
+ bool macroSkew() const { return macroSkew_; }
void setBufferListInferred(bool inferred) { bufferListInferred_ = inferred; }
bool isBufferListInferred() const { return bufferListInferred_; }
void setSinkBufferInferred(bool inferred) { sinkBufferInferred_ = inferred; }
@@ -390,6 +392,7 @@ class CtsOptions : public odb::dbBlockCallBackObj
stt::SteinerTreeBuilder* sttBuilder_ = nullptr;
bool obsAware_ = true;
bool insertionDelay_ = true;
+ bool macroSkew_ = true;
bool bufferListInferred_ = false;
bool sinkBufferInferred_ = false;
bool rootBufferInferred_ = false;
diff --git a/src/cts/src/LatencyBalancer.cpp b/src/cts/src/LatencyBalancer.cpp
index c422ce7521..6e87278e51 100644
--- a/src/cts/src/LatencyBalancer.cpp
+++ b/src/cts/src/LatencyBalancer.cpp
@@ -4,6 +4,7 @@
#include "LatencyBalancer.h"
#include <algorithm>
+#include <cassert>
#include <cmath>
#include <limits>
#include <map>
@@ -16,6 +17,7 @@
#include "Clock.h"
#include "CtsOptions.h"
+#include "MacroLatencyBalance.h"
#include "TreeBuilder.h"
#include "Util.h"
#include "cts/TritonCTS.h"
@@ -27,15 +29,21 @@
#include "odb/geom.h"
#include "sta/Clock.hh"
#include "sta/Delay.hh"
+#include "sta/ExceptionPath.hh"
#include "sta/Graph.hh"
#include "sta/GraphDelayCalc.hh"
#include "sta/Liberty.hh"
+#include "sta/MinMax.hh"
#include "sta/Mode.hh"
#include "sta/NetworkClass.hh"
#include "sta/Path.hh"
#include "sta/PathEnd.hh"
#include "sta/PathExpanded.hh"
+#include "sta/Search.hh"
+#include "sta/SearchClass.hh"
#include "sta/Sdc.hh"
+#include "sta/Sta.hh"
+#include "sta/StringUtil.hh"
#include "sta/TimingArc.hh"
#include "sta/TimingModel.hh"
#include "utl/Logger.h"
@@ -45,6 +53,12 @@ namespace cts {
using utl::CTS;
int LatencyBalancer::run()
+{
+ prepare();
+ return balance(worseDelay_);
+}
+
+float LatencyBalancer::prepare()
{
logger_->info(CTS,
33,
@@ -53,6 +67,16 @@ int LatencyBalancer::run()
initSta();
findLeafBuilders(root_);
buildGraph(root_->getTopInputNet());
+ return worseDelay_;
+}
+
+int LatencyBalancer::balance(float target_worse_delay)
+{
+ // Pad every sink up to the shared global target so a macro sink with
+ // internal clock-tree-path delay D lands at (target - D) -- i.e. its clock
+ // pin is delivered D earlier than a plain register, cancelling the macro's
+ // late internal capture instead of leaving it as a boundary hold violation.
+ worseDelay_ = target_worse_delay;
bufferDelay_ = computeBufferDelay(0);
balanceLatencies(0);
logger_->info(CTS,
@@ -178,9 +202,13 @@ void LatencyBalancer::buildGraph(odb::dbNet* clkInputNet)
if (inst2builder_.find(sinkName) != inst2builder_.end()) {
auto builder = inst2builder_[sinkName];
- float builerAvgArrival = computeAveSinkArrivals(builder);
- worseDelay_ = std::max(worseDelay_, builerAvgArrival);
- graph_[sinkId].arrival = builerAvgArrival;
+ // Contribute the sub-tree's WORST (deepest) sink, not its average, so
+ // the shared global target reflects the deepest launch register and
+ // macro clock pins are delivered early relative to it (fixing the
+ // macro-boundary hold). Averaging understated a deep register tree.
+ float builderMaxArrival = computeMaxSinkArrivals(builder);
+ worseDelay_ = std::max(worseDelay_, builderMaxArrival);
+ graph_[sinkId].arrival = builderMaxArrival;
continue;
}
@@ -229,6 +257,282 @@ void LatencyBalancer::buildGraph(odb::dbNet* clkInputNet)
}
}
+odb::dbInst* LatencyBalancer::instOf(const sta::Pin* pin) const
+{
+ // Resolve via the pin's INSTANCE, not the pin's dbITerm: a hierarchical
+ // macro's path pin can present as a dbModITerm (staToDb(pin) then yields no
+ // dbITerm), but the instance always resolves to the underlying dbInst.
+ sta::Instance* inst = network_->instance(pin);
+ return inst ? network_->staToDb(inst) : nullptr;
+}
+
+void LatencyBalancer::collectMacros()
+{
+ if (!macroTiming_.empty()) {
+ return;
+ }
+ Clock clock = root_->getClock();
+ clock.forEachSink([&](const ClockInst& sink) {
+ odb::dbITerm* clkIterm = sink.getDbInputPin();
+ if (!clkIterm) {
+ return;
+ }
+ odb::dbInst* inst = clkIterm->getInst();
+ if (!inst) {
+ return;
+ }
+ bool isMacro = inst->isBlock();
+ if (!isMacro) {
+ sta::LibertyCell* libCell
+ = network_->libertyCell(network_->dbToSta(inst));
+ odb::dbMTerm* mterm = clkIterm->getMTerm();
+ if (libCell && mterm) {
+ sta::LibertyPort* libPort
+ = libCell->findLibertyPort(mterm->getConstName());
+ if (libPort
+ && (libPort->clkTreeDelay(0.0, sta::RiseFall::rise(),
+ sta::MinMax::max())
+ != 0.0
+ || libPort->clkTreeDelay(0.0, sta::RiseFall::fall(),
+ sta::MinMax::max())
+ != 0.0)) {
+ isMacro = true;
+ }
+ }
+ }
+ if (!isMacro) {
+ return;
+ }
+ auto it = macroInst2timing_.find(inst);
+ if (it != macroInst2timing_.end()) {
+ macroTiming_[it->second].clkIterms.push_back(clkIterm);
+ return;
+ }
+ MacroTiming mt;
+ mt.inst = inst;
+ mt.clkIterms.push_back(clkIterm);
+ macroInst2timing_[inst] = (int) macroTiming_.size();
+ macroTiming_.push_back(mt);
+ });
+}
+
+void LatencyBalancer::gatherMacroTiming()
+{
+ if (macroTiming_.empty()) {
+ return;
+ }
+ gatherCaptureSlacks();
+ gatherLaunchSlacks();
+}
+
+void LatencyBalancer::gatherCaptureSlacks()
+{
+ // Capture side: the macro is the ENDPOINT. Worst hold/setup slack over its
+ // data input pins governs whether a core register's launch races into (hold)
+ // or fails to reach (setup) the macro.
+ for (MacroTiming& mt : macroTiming_) {
+ for (odb::dbITerm* iterm : mt.inst->getITerms()) {
+ if (iterm->getIoType() == odb::dbIoType::OUTPUT) {
+ continue;
+ }
+ sta::Pin* p = network_->dbToSta(iterm);
+ if (!p) {
+ continue;
+ }
+ if (openSta_->isClock(p, openSta_->cmdMode())) {
+ continue;
+ }
+ sta::Vertex* v = timingGraph_->pinLoadVertex(p);
+ if (!v) {
+ continue;
+ }
+ mt.s_cap_hold = std::min(
+ mt.s_cap_hold, (double) openSta_->slack(v, sta::MinMax::min()));
+ }
+ }
+}
+
+void LatencyBalancer::gatherLaunchSlacks()
+{
+ // Launch side: the macro is the STARTPOINT. A macro whose read launches too
+ // early shows up as a HOLD-violated ENDPOINT (the downstream register) whose
+ // worst path STARTS at the macro. Iterate hold-violated endpoints (cheap,
+ // incrementally maintained -- the RepairHold pattern) and bin each one's
+ // worst-path startpoint back to its launching macro. This deliberately avoids
+ // findPathEnds with an ExceptionFrom filter: that leaves filtered-arrival tags
+ // referencing graph vertices, which the subsequent buffer insertion
+ // invalidates -> a later search/estimate_parasitics dereferences stale tags
+ // and crashes. Per-vertex worst-slack paths carry no such filter state.
+ openSta_->searchPreamble();
+ openSta_->ensureLevelized();
+ sta::VertexSet& ends = openSta_->search()->endpoints();
+ int nViol = 0, nBinned = 0;
+ for (sta::Vertex* end : ends) {
+ if (openSta_->isClock(end->pin(), openSta_->cmdMode())) {
+ continue;
+ }
+ const double hold = (double) openSta_->slack(end, sta::MinMax::min());
+ if (hold >= 0.0) {
+ continue;
+ }
+ ++nViol;
+ sta::Path* wp = openSta_->vertexWorstSlackPath(end, sta::MinMax::min());
+ if (!wp) {
+ continue;
+ }
+ // Find the launching macro on the path. It is the startpoint, but scan
+ // forward for the first macro instance so a startpoint pin that doesn't
+ // resolve cleanly still attributes the violation to the right macro.
+ sta::PathExpanded ex(wp, openSta_);
+ int timingIdx = -1;
+ for (size_t i = ex.startIndex(); i < ex.size(); ++i) {
+ odb::dbInst* di = instOf(ex.path(i)->vertex(openSta_)->pin());
+ if (di) {
+ auto it = macroInst2timing_.find(di);
+ if (it != macroInst2timing_.end()) {
+ timingIdx = it->second;
+ break;
+ }
+ }
+ }
+ if (timingIdx < 0) {
+ continue;
+ }
+ ++nBinned;
+ macroTiming_[timingIdx].s_launch_hold
+ = std::min(macroTiming_[timingIdx].s_launch_hold, hold);
+ }
+ debugPrint(logger_, CTS, "macro skew", 1,
+ "launch gather: {} hold-violated endpoints, {} binned to macros",
+ nViol, nBinned);
+}
+
+int LatencyBalancer::skewMacros()
+{
+ collectMacros();
+ debugPrint(logger_, CTS, "macro skew", 1,
+ "skewMacros clock {}: {} macros tracked",
+ root_->getClock().getSdcName(), macroTiming_.size());
+ if (macroTiming_.empty()) {
+ return 0;
+ }
+ // Corrective push runs AFTER base balancing, so the register counterparts are
+ // at the global target and the macros at (global - insDelay) -- the final
+ // operating point at which the measured launch/capture hold slacks are valid.
+ // Each pass pushes launch-hold-violated macros later (bounded by capture-hold
+ // room); iterate a few times so macro<->macro coupling and quantization
+ // settle, stopping when nothing more is inserted.
+ assert(bufferDelay_ > 0);
+ int totalInserted = 0;
+ for (int iter = 0; iter < macroSkewIters_; ++iter) {
+ // Reflect the buffers inserted so far, then measure ALL macros before
+ // mutating any -- the PathEndSeq from findPathEnds is invalidated by the
+ // first buffer insertion.
+ openSta_->updateTiming(false);
+ for (MacroTiming& mt : macroTiming_) {
+ mt.s_cap_hold = mt.s_launch_hold
+ = std::numeric_limits<float>::infinity();
+ }
+ gatherMacroTiming();
+ debugPrint(logger_, CTS, "macro skew", 1, "iter {}: tracking {} macros",
+ iter, macroTiming_.size());
+
+ int iterInserted = 0;
+ double maxDelta = 0.0;
+ for (const MacroTiming& mt : macroTiming_) {
+ const MacroSkewInput in{
+ mt.s_cap_hold, mt.s_launch_hold, bufferDelay_, /*hold_margin=*/0.0};
+ const MacroSkewResult r = computeMacroSkew(in);
+ if (r.num_buffers > 0) {
+ debugPrint(logger_, CTS, "macro skew", 2,
+ "{} scap_h={:.4e} slaunch_h={:.4e} delta={:.4e} nbuf={}",
+ mt.inst->getName(), mt.s_cap_hold, mt.s_launch_hold,
+ r.delta_realized, r.num_buffers);
+ // Push every clock pin of the macro (read + write for a dual-port
+ // memory) by the same amount, so its internal timing shifts coherently.
+ for (odb::dbITerm* clkIterm : mt.clkIterms) {
+ insertMacroSkewBuffers(clkIterm, r.num_buffers);
+ }
+ iterInserted += r.num_buffers;
+ maxDelta = std::max(maxDelta, r.delta_realized);
+ }
+ }
+ totalInserted += iterInserted;
+ if (iterInserted == 0 || maxDelta < bufferDelay_ / 2.0) {
+ break;
+ }
+ }
+ return totalInserted;
+}
+
+void LatencyBalancer::insertMacroSkewBuffers(odb::dbITerm* clkIterm,
+ int numBuffers)
+{
+ if (numBuffers <= 0 || clkIterm == nullptr) {
+ return;
+ }
+ odb::dbNet* drivingNet = clkIterm->getNet();
+ if (drivingNet == nullptr) {
+ return;
+ }
+
+ int srcX, srcY;
+ odb::dbITerm* driver = drivingNet->getFirstOutput();
+ if (driver != nullptr) {
+ driver->getAvgXY(&srcX, &srcY);
+ } else {
+ clkIterm->getAvgXY(&srcX, &srcY);
+ }
+ int sinkX, sinkY;
+ clkIterm->getAvgXY(&sinkX, &sinkY);
+ const float offsetX = (float) (sinkX - srcX) / (numBuffers + 1);
+ const float offsetY = (float) (sinkY - srcY) / (numBuffers + 1);
+
+ odb::dbMaster* bufferMaster
+ = db_->findMaster(options_->getRootBuffer().c_str());
+ clkIterm->disconnect();
+
+ for (int i = 0; i < numBuffers; i++) {
+ const double locX = (double) (srcX + offsetX * (i + 1)) / wireSegmentUnit_;
+ const double locY = (double) (srcY + offsetY * (i + 1)) / wireSegmentUnit_;
+ Point<double> bufferLoc(locX, locY);
+ Point<double> legalBufferLoc
+ = root_->legalizeOneBuffer(bufferLoc, options_->getRootBuffer());
+ odb::Point loc{static_cast<int>(legalBufferLoc.getX() * wireSegmentUnit_),
+ static_cast<int>(legalBufferLoc.getY() * wireSegmentUnit_)};
+
+ const std::string clkName = root_->getClock().getSdcName();
+ const std::string newNetName
+ = fmt::format("macroskewnet_{}_{}", delayBufIndex_, clkName);
+ const std::string newBufferName
+ = fmt::format("macroskewbuf_{}_{}", delayBufIndex_++, clkName);
+
+ odb::PtrSet<odb::dbObject> load_pins;
+ load_pins.insert(clkIterm);
+ const bool loads_on_different_nets = true;
+ odb::dbInst* buffer = drivingNet->insertBufferBeforeLoads(
+ load_pins,
+ bufferMaster,
+ &loc,
+ newBufferName.c_str(),
+ newNetName.c_str(),
+ odb::dbNameUniquifyType::IF_NEEDED,
+ loads_on_different_nets);
+
+ debugPrint(logger_,
+ CTS,
+ "macro skew",
+ 1,
+ "macro skew buffer {} inserted at ({} {})",
+ buffer->getName(),
+ loc.getX(),
+ loc.getY());
+
+ odb::dbITerm* drvrPin = buffer->getFirstOutput();
+ drivingNet = drvrPin->getNet();
+ }
+}
+
odb::dbITerm* LatencyBalancer::getFirstInput(odb::dbInst* inst) const
{
odb::dbSet<odb::dbITerm> iterms = inst->getITerms();
@@ -298,9 +602,11 @@ float LatencyBalancer::computeAveSinkArrivals(TreeBuilder* builder)
// compute average input arrival at all sinks
float sumArrivals = 0.0;
unsigned numSinks = 0;
+ float maxArrival = 0.0;
clock.forEachSink([&](const ClockInst& sink) {
odb::dbITerm* iterm = sink.getDbInputPin();
- computeSinkArrivalRecur(topInputClockNet, iterm, sumArrivals, numSinks);
+ computeSinkArrivalRecur(
+ topInputClockNet, iterm, sumArrivals, numSinks, maxArrival);
});
float aveArrival = 0.0;
if (numSinks) {
@@ -320,10 +626,26 @@ float LatencyBalancer::computeAveSinkArrivals(TreeBuilder* builder)
return aveArrival;
}
+float LatencyBalancer::computeMaxSinkArrivals(TreeBuilder* builder)
+{
+ Clock clock = builder->getClock();
+ odb::dbNet* topInputClockNet = builder->getTopInputNet();
+ float sumArrivals = 0.0;
+ unsigned numSinks = 0;
+ float maxArrival = 0.0;
+ clock.forEachSink([&](const ClockInst& sink) {
+ odb::dbITerm* iterm = sink.getDbInputPin();
+ computeSinkArrivalRecur(
+ topInputClockNet, iterm, sumArrivals, numSinks, maxArrival);
+ });
+ return maxArrival;
+}
+
void LatencyBalancer::computeSinkArrivalRecur(odb::dbNet* topClokcNet,
odb::dbITerm* iterm,
float& sumArrivals,
- unsigned& numSinks)
+ unsigned& numSinks,
+ float& maxArrival)
{
if (iterm) {
odb::dbInst* inst = iterm->getInst();
@@ -357,6 +679,7 @@ void LatencyBalancer::computeSinkArrivalRecur(odb::dbNet* topClokcNet,
}
}
sumArrivals += (arrival + insDelay);
+ maxArrival = std::max(maxArrival, arrival + insDelay);
numSinks++;
}
} else {
@@ -372,7 +695,7 @@ void LatencyBalancer::computeSinkArrivalRecur(odb::dbNet* topClokcNet,
odb::dbITerm* inTerm = *iter;
if (inTerm->getIoType() == odb::dbIoType::INPUT) {
computeSinkArrivalRecur(
- topClokcNet, inTerm, sumArrivals, numSinks);
+ topClokcNet, inTerm, sumArrivals, numSinks, maxArrival);
}
}
}
diff --git a/src/cts/src/LatencyBalancer.h b/src/cts/src/LatencyBalancer.h
index ba121f0f42..c29a27d6f7 100644
--- a/src/cts/src/LatencyBalancer.h
+++ b/src/cts/src/LatencyBalancer.h
@@ -12,8 +12,10 @@
#include "Clock.h"
#include "CtsOptions.h"
+#include "MacroLatencyBalance.h"
#include "TreeBuilder.h"
#include "Util.h"
+#include "odb/PtrSetMap.h"
#include "odb/db.h"
#include "sta/Delay.hh"
#include "utl/Logger.h"
@@ -24,6 +26,7 @@ class dbNetwork;
class LibertyCell;
class Vertex;
class Graph;
+class Pin;
} // namespace sta
namespace cts {
@@ -41,6 +44,22 @@ struct GraphNode
double arrival = 0.0;
int nBuffInsert = -1;
odb::dbITerm* inputTerm = nullptr;
+ bool isMacro = false;
+ odb::dbInst* macroInst = nullptr;
+};
+
+// Per-macro worst slacks feeding computeMacroSkew. Gathered before any netlist
+// mutation; +inf means no such timing path constrains this side.
+struct MacroTiming
+{
+ odb::dbInst* inst = nullptr;
+ // All of the macro's clock input pins. A dual-port memory has a read clock
+ // and a write clock; the read clock drives the launch path, the write clock
+ // the capture path, so BOTH must be pushed by the same delta to shift the
+ // macro's internal timing (delaying only one leaves the other's paths unfixed).
+ std::vector<odb::dbITerm*> clkIterms;
+ double s_cap_hold = std::numeric_limits<float>::infinity();
+ double s_launch_hold = std::numeric_limits<float>::infinity();
};
class LatencyBalancer
@@ -67,21 +86,50 @@ class LatencyBalancer
}
int run();
+ // Two-phase entry points so several balancers (e.g. the macro tree and the
+ // register tree) can share one global target latency instead of each
+ // balancing to its own local worst. prepare() builds the graph and returns
+ // this tree's local worst effective latency (clock arrival + macro insertion
+ // delay); balance() pads every sink up to the supplied target.
+ float prepare();
+ int balance(float target_worse_delay);
+ // Post-base-balance corrective push: with registers already at the global
+ // target and macros at (global - insDelay), measure each macro's worst
+ // capture/launch hold+setup slack and add delay buffers on its clock branch
+ // to push it LATER by the timing-driven computeMacroSkew() amount. Must run
+ // AFTER balance() so bufferDelay_ is set and the operating point is final.
+ int skewMacros();
private:
void initSta();
void findLeafBuilders(TreeBuilder* builder);
void buildGraph(odb::dbNet* clkInputNet);
+ // Measure every macro sink's worst capture- and launch-side hold/setup slack
+ // (all reads happen here, before any buffer is inserted, so slacks are valid).
+ // Enumerate the macro clock sinks directly from the macro tree's clock (a
+ // clustered macro tree folds its sinks into leaf-builder aggregates in
+ // buildGraph, so they are invisible there -- forEachSink still sees them all).
+ void collectMacros();
+ void gatherMacroTiming();
+ void gatherCaptureSlacks();
+ void gatherLaunchSlacks();
+ void insertMacroSkewBuffers(odb::dbITerm* clkIterm, int numBuffers);
+ odb::dbInst* instOf(const sta::Pin* pin) const;
odb::dbITerm* getFirstInput(odb::dbInst* inst) const;
float getVertexClkArrival(sta::Vertex* sinkVertex,
odb::dbNet* topNet,
odb::dbITerm* iterm);
sta::ArcDelay computeBufferDelay(double extra_out_cap);
float computeAveSinkArrivals(TreeBuilder* builder);
+ // Worst (max) effective sink latency (clock arrival + macro insertion delay)
+ // over the builder's sinks. Used so a register sub-tree contributes its
+ // deepest register to the shared global target, not its average.
+ float computeMaxSinkArrivals(TreeBuilder* builder);
void computeSinkArrivalRecur(odb::dbNet* topClokcNet,
odb::dbITerm* iterm,
float& sumArrivals,
- unsigned& numSinks);
+ unsigned& numSinks,
+ float& maxArrival);
void computeNumberOfDelayBuffers(int nodeId, int srcX, int srcY);
// DFS search throw the tree graph to insert delay buffers. At each node,
@@ -113,6 +161,9 @@ class LatencyBalancer
int delayBufIndex_{0};
std::vector<GraphNode> graph_;
std::map<std::string, TreeBuilder*> inst2builder_;
+ std::vector<MacroTiming> macroTiming_;
+ odb::PtrMap<odb::dbInst, int> macroInst2timing_;
+ int macroSkewIters_ = 8;
};
} // namespace cts
diff --git a/src/cts/src/MacroLatencyBalance.cc b/src/cts/src/MacroLatencyBalance.cc
new file mode 100644
index 0000000000..3929ec13d5
--- /dev/null
+++ b/src/cts/src/MacroLatencyBalance.cc
@@ -0,0 +1,67 @@
+// SPDX-License-Identifier: BSD-3-Clause
+// Copyright (c) 2019-2025, The OpenROAD Authors
+
+#include "MacroLatencyBalance.h"
+
+#include <algorithm>
+#include <cmath>
+#include <vector>
+
+namespace cts {
+
+std::vector<double> computeSinkTargets(const std::vector<SinkLatency>& sinks)
+{
+ std::vector<double> targets;
+ if (sinks.empty()) {
+ return targets;
+ }
+
+ // Global worst effective capture latency across every sink (macros and
+ // registers together), at the min (hold) corner.
+ double global = sinks.front().arrival + sinks.front().ins_delay_min;
+ for (const SinkLatency& sink : sinks) {
+ const double effective = sink.arrival + sink.ins_delay_min;
+ if (effective > global) {
+ global = effective;
+ }
+ }
+
+ // Deliver each sink's clock so its effective capture aligns at the global:
+ // a macro (ins_delay_min > 0) lands ins_delay_min earlier than a register.
+ targets.reserve(sinks.size());
+ for (const SinkLatency& sink : sinks) {
+ targets.push_back(global - sink.ins_delay_min);
+ }
+ return targets;
+}
+
+MacroSkewResult computeMacroSkew(const MacroSkewInput& in)
+{
+ MacroSkewResult r{};
+ r.delta_needed = in.hold_margin - in.s_launch_hold;
+ r.capture_room = in.s_cap_hold - in.hold_margin;
+
+ // Only a launch-hold violation (delta_needed > 0) is fixable by pushing later.
+ // No launch path (s_launch_hold = +inf -> delta_needed = -inf) or launch
+ // already met -> no push.
+ int num_buffers = 0;
+ if (std::isfinite(r.delta_needed) && r.delta_needed > 0.0) {
+ // Round UP so the push actually clears the violation...
+ num_buffers = (int) std::ceil(r.delta_needed / in.buffer_delay);
+ // ... but never push capture hold below margin (capture_room may be +inf).
+ if (std::isfinite(r.capture_room)) {
+ const int cap = (int) std::floor(r.capture_room / in.buffer_delay);
+ if (num_buffers > cap) {
+ num_buffers = cap;
+ }
+ }
+ }
+ if (num_buffers < 0) {
+ num_buffers = 0;
+ }
+ r.num_buffers = num_buffers;
+ r.delta_realized = num_buffers * in.buffer_delay;
+ return r;
+}
+
+} // namespace cts
diff --git a/src/cts/src/MacroLatencyBalance.h b/src/cts/src/MacroLatencyBalance.h
new file mode 100644
index 0000000000..02c3cefa1e
--- /dev/null
+++ b/src/cts/src/MacroLatencyBalance.h
@@ -0,0 +1,88 @@
+// SPDX-License-Identifier: BSD-3-Clause
+// Copyright (c) 2019-2025, The OpenROAD Authors
+
+#pragma once
+
+#include <vector>
+
+namespace cts {
+
+// One clock sink for macro-vs-register latency balancing.
+struct SinkLatency
+{
+ // Clock arrival at the sink's clock pin (the top-level clock-tree insertion
+ // delay reaching the pin).
+ double arrival;
+ // The sink's own internal clock-tree insertion delay from its clock pin to
+ // its internal registers, at the two corners of its Liberty
+ // min/max_clock_tree_path. Both are 0 for a plain register that captures at
+ // its clock pin.
+ //
+ // ins_delay_min is the hold corner and ins_delay_max the setup corner. A
+ // macro launches read data through its shortest internal clock path, so
+ // ins_delay_min governs whether that data races into a downstream register
+ // (boundary hold). ins_delay_max governs setup.
+ double ins_delay_min;
+ double ins_delay_max;
+};
+
+// Target clock-pin arrival for each sink so that every sink's EFFECTIVE capture
+// time (arrival + insertion delay) aligns at the global worst effective latency
+// across ALL sinks. A macro with insertion delay D is delivered its clock at
+// target = global - D, i.e. D EARLIER than a plain register, cancelling its late
+// internal capture instead of leaving it as a boundary hold violation.
+//
+// The offset used is the MIN corner (min_clock_tree_path). Using the MAX corner
+// over-pulls a macro whose min_clock_tree_path < max_clock_tree_path: its
+// internal registers then LAUNCH earlier than global, so read data races into a
+// downstream core register and the boundary HOLD is violated (in our design this
+// shifted the worst path onto an SRAM read). Compensating at the min corner
+// lands each macro's internal launch/capture at ~global -- a zero-skew
+// equivalent -- so the boundary hold is set by the data path, not the clock.
+//
+// The global is taken across macros AND registers together, so a deep register
+// tree still pulls every macro clock pin its own insertion delay earlier than
+// that register rather than each tree balancing to its own local worst.
+std::vector<double> computeSinkTargets(const std::vector<SinkLatency>& sinks);
+
+// Timing-driven per-macro clock skew schedule (Fishburn useful-skew specialised
+// to one macro clock pin). A macro's single clock arrival trades its two HOLD
+// directions against each other: shifting the pin LATER by delta moves the
+// capture-side hold (core register -> macro) DOWN by delta and the launch-side
+// hold (macro -> core register) UP by delta. So a launch-dominated macro (e.g.
+// an SRAM read whose data races into a downstream register) is fixed by pushing
+// its clock LATER, which a fixed "pull every macro earlier by its Liberty
+// insertion delay" rule can never do.
+//
+// This is a HOLD-only pass: push a macro just far enough to clear a launch-hold
+// violation, capped by its capture-hold room so the fix never drives capture
+// hold negative. Setup is deliberately ignored -- at CTS setup is not yet
+// optimized (repair_timing owns it post-route), so reacting to CTS-time setup
+// would trigger huge, spurious pushes.
+struct MacroSkewInput
+{
+ // Worst HOLD slack over the macro's capture paths (macro = endpoint) and
+ // launch paths (macro = startpoint). +inf means "no such path".
+ double s_cap_hold;
+ double s_launch_hold;
+ double buffer_delay; // one root-buffer delay (> 0): the quantization grain
+ double hold_margin; // hold slack floor to reach / preserve
+};
+
+struct MacroSkewResult
+{
+ double delta_needed; // hold_margin - s_launch_hold (> 0 iff launch hold violated)
+ double capture_room; // s_cap_hold - hold_margin (max push before capture hold breaks)
+ double delta_realized; // num_buffers * buffer_delay (>= 0, add-only)
+ int num_buffers; // delay buffers to add on the macro's clock branch
+};
+
+// Realization is add-only (delay buffers only ADD latency): only a launch-hold
+// violation yields a positive push. A capture-hold violation "wants earlier",
+// which is unrealizable here (0 buffers) -- it is left to the register tree
+// sitting at the global, or to data-path hold repair. The push is rounded UP to
+// clear the violation, then capped so it never pushes capture hold below margin
+// (if both sides are violated, capture_room <= 0 -> 0 buffers).
+MacroSkewResult computeMacroSkew(const MacroSkewInput& in);
+
+} // namespace cts
diff --git a/src/cts/src/TritonCTS.cpp b/src/cts/src/TritonCTS.cpp
index c27f84b051..51d8f3e1a6 100644
--- a/src/cts/src/TritonCTS.cpp
+++ b/src/cts/src/TritonCTS.cpp
@@ -2620,18 +2620,41 @@ void TritonCTS::balanceMacroRegisterLatencies()
double capPerDBU = estimate_parasitics_->wireClkCapacitance(corner) * 1e-6
/ block_->getDbUnitsPerMicron();
+ est::IncrementalParasiticsGuard parasitics_guard(estimate_parasitics_);
+ // Pass 1: build every root tree's graph and find the GLOBAL worst effective
+ // latency (clock arrival + macro insertion delay) across all trees. Balancing
+ // every tree to this shared target -- instead of each tree to its own local
+ // worst -- is what delivers a macro clock pin early relative to the register
+ // tree. Balancing per-tree left macro clock pins
+ // LATER than the register tree, so their internal insertion delay showed up
+ // as a large boundary hold violation.
+ std::vector<std::unique_ptr<LatencyBalancer>> balancers;
+ float globalWorseDelay = std::numeric_limits<float>::min();
for (auto& builder : std::ranges::reverse_view(builders_)) {
if (builder->getParent() == nullptr && !builder->getChildren().empty()) {
- est::IncrementalParasiticsGuard parasitics_guard(estimate_parasitics_);
- LatencyBalancer balancer = LatencyBalancer(builder.get(),
- options_,
- logger_,
- db_,
- network_,
- openSta_,
- techChar_->getLengthUnit(),
- capPerDBU);
- totalDelayBuff += balancer.run();
+ auto balancer = std::make_unique<LatencyBalancer>(builder.get(),
+ options_,
+ logger_,
+ db_,
+ network_,
+ openSta_,
+ techChar_->getLengthUnit(),
+ capPerDBU);
+ globalWorseDelay = std::max(globalWorseDelay, balancer->prepare());
+ balancers.push_back(std::move(balancer));
+ }
+ }
+ // Pass 2: pad every tree up to the shared global target.
+ for (auto& balancer : balancers) {
+ totalDelayBuff += balancer->balance(globalWorseDelay);
+ }
+ // Pass 3: timing-driven corrective push. With every tree now base-balanced
+ // (registers at global, macros at global - insDelay), measure each macro's
+ // worst capture/launch hold+setup slack at that final operating point and add
+ // delay buffers on its clock branch to push it later by computeMacroSkew().
+ if (options_->macroSkew()) {
+ for (auto& balancer : balancers) {
+ totalDelayBuff += balancer->skewMacros();
}
}
if (totalDelayBuff) {
diff --git a/src/cts/test/BUILD b/src/cts/test/BUILD
index 71a32a67e7..5a04547c18 100644
--- a/src/cts/test/BUILD
+++ b/src/cts/test/BUILD
@@ -248,6 +248,19 @@ cc_test(
],
)
+# Tier-1 pure-kernel unit test (docs/agents/testing-strategy.md): deps ONLY the
+# dependency-free macro-latency kernel + gtest, so the edit/build/run loop stays
+# sub-second (no libcts/odb/sta relink).
+cc_test(
+ name = "macro_latency_balance_test",
+ srcs = ["macro_latency_balance_test.cc"],
+ deps = [
+ "//src/cts:macro_latency_balance",
+ "@googletest//:gtest",
+ "@googletest//:gtest_main",
+ ],
+)
+
py_test(
name = "cts_man_tcl_check",
srcs = ["cts_man_tcl_check.py"],
diff --git a/src/cts/test/macro_latency_balance_test.cc b/src/cts/test/macro_latency_balance_test.cc
new file mode 100644
index 0000000000..735bf61fbc
--- /dev/null
+++ b/src/cts/test/macro_latency_balance_test.cc
@@ -0,0 +1,201 @@
+// SPDX-License-Identifier: BSD-3-Clause
+// Copyright (c) 2019-2025, The OpenROAD Authors
+
+#include <algorithm>
+#include <limits>
+#include <vector>
+
+#include "MacroLatencyBalance.h"
+#include "gtest/gtest.h"
+
+namespace cts {
+
+namespace {
+constexpr double kInf = std::numeric_limits<double>::infinity();
+} // namespace
+
+// Regression pin for the macro-clock-latency compensation bug: a macro sink with
+// internal clock-tree insertion delay must be delivered its clock EARLIER than a
+// plain register by that insertion delay, so its late internal capture is
+// cancelled instead of surfacing as a boundary hold violation.
+TEST(MacroLatencyBalance, MacroDeliveredEarlierByInsDelay)
+{
+ // 3 registers (~900 ps arrival, no internal delay) + 1 macro slave whose raw
+ // clock arrival is 800 ps but which has 150 ps of internal clock-tree-path
+ // delay.
+ const std::vector<SinkLatency> sinks = {{900.0, 0.0, 0.0},
+ {880.0, 0.0, 0.0},
+ {910.0, 0.0, 0.0},
+ {800.0, 150.0, 150.0}};
+ const std::vector<double> t = computeSinkTargets(sinks);
+
+ // Global effective latency = max(900, 880, 910, 800 + 150 = 950) = 950.
+ EXPECT_DOUBLE_EQ(t[0], 950.0); // register delivered at the global
+ EXPECT_DOUBLE_EQ(t[3], 800.0); // macro delivered 150 ps earlier
+ EXPECT_LT(t[3], t[0]); // macro clock EARLIER than registers
+ EXPECT_DOUBLE_EQ(t[0] - t[3], 150.0); // ... by exactly its insertion delay
+}
+
+// The global target must be SHARED across macros and registers: when a deep
+// register dominates, the macro is still delivered its insertion delay earlier
+// than that register. The bug balanced the macro tree to its own local worst
+// (excluding the register tree), so the macro ended up later than registers.
+TEST(MacroLatencyBalance, GlobalIsSharedAcrossMacrosAndRegisters)
+{
+ const std::vector<SinkLatency> sinks = {{1080.0, 0.0, 0.0},
+ {800.0, 150.0, 150.0}};
+ const std::vector<double> t = computeSinkTargets(sinks);
+
+ EXPECT_DOUBLE_EQ(t[0], 1080.0); // deep register sets the global
+ EXPECT_DOUBLE_EQ(t[1], 930.0); // macro delivered 1080 - 150
+ EXPECT_DOUBLE_EQ(t[0] - t[1], 150.0); // macro earlier by its insertion delay
+}
+
+// The exact failure we hit: a launch-side macro (SRAM read) whose internal clock
+// path is asymmetric (min_clock_tree_path < max_clock_tree_path). Compensation
+// must use the MIN corner. Using the MAX corner over-pulls the clock pin, so the
+// macro's internal registers launch EARLIER than the global and read data races
+// into the downstream core register -- a boundary HOLD violation of (max - min).
+// This is what shifted our worst path from the macro slave onto an SRAM.
+TEST(MacroLatencyBalance, LaunchSideMacroUsesMinCornerNotMax)
+{
+ // SRAM read macro: 97 ps min / 180 ps max internal clock-tree path.
+ const std::vector<SinkLatency> sinks = {{900.0, 0.0, 0.0}, // register
+ {1080.0, 0.0, 0.0}, // deep register
+ {800.0, 97.0, 180.0}}; // SRAM read
+ const std::vector<double> t = computeSinkTargets(sinks);
+
+ // global = max(900, 1080, 800 + 97 = 897) = 1080.
+ const double global = 1080.0;
+ const double sram_min = 97.0;
+ const double sram_max = 180.0;
+
+ // Compensated at the MIN corner: pin delivered global - 97 = 983.
+ EXPECT_DOUBLE_EQ(t[2], global - sram_min);
+ // NOT at the max corner (that would be 900 -- the over-pull).
+ EXPECT_NE(t[2], global - sram_max);
+
+ // The macro's internal launch fires at pin + min = 983 + 97 = 1080 = global:
+ // aligned, hold-safe. The max-corner over-pull would launch at 900 + 97 = 997,
+ // i.e. (max - min) = 83 ps EARLY -- the boundary hold violation.
+ EXPECT_DOUBLE_EQ(t[2] + sram_min, global);
+ EXPECT_DOUBLE_EQ((global - sram_max) + sram_min, global - (sram_max - sram_min));
+}
+
+// Each macro is compensated by its OWN insertion delay, not a single shared
+// magnitude: a macro slave (~150 ps) and an SRAM (~97 ps) sharing one clock domain
+// land at different targets, both = global - their own min offset.
+TEST(MacroLatencyBalance, PerMacroDistinctOffsets)
+{
+ const std::vector<SinkLatency> sinks = {{1080.0, 0.0, 0.0}, // deep register
+ {850.0, 150.0, 180.0}, // macro slave
+ {800.0, 97.0, 180.0}}; // SRAM read
+ const std::vector<double> t = computeSinkTargets(sinks);
+
+ // global = max(1080, 850 + 150 = 1000, 800 + 97 = 897) = 1080.
+ EXPECT_DOUBLE_EQ(t[0], 1080.0); // register at the global
+ EXPECT_DOUBLE_EQ(t[1], 1080.0 - 150.0); // macro slave: global - its own 150
+ EXPECT_DOUBLE_EQ(t[2], 1080.0 - 97.0); // SRAM: global - its own 97
+ EXPECT_NE(t[1], t[2]); // distinct, per-sink -- not shared
+}
+
+TEST(MacroLatencyBalance, EmptyInputIsEmpty)
+{
+ EXPECT_TRUE(computeSinkTargets({}).empty());
+}
+
+// The exact regression: a launch-dominated macro (SRAM read, hold
+// -155 ps) is pushed LATER to clear the launch-hold violation -- a fixed
+// pull-early rule could never do this. Here capture room (100 ps) is the binding
+// limit, so the push stops when capture hold reaches margin (both sides ~equal).
+TEST(MacroSkew, LaunchDominatedSramPushedLater)
+{
+ const MacroSkewResult r = computeMacroSkew({.s_cap_hold = 0.100,
+ .s_launch_hold = -0.155,
+ .buffer_delay = 0.020,
+ .hold_margin = 0.0});
+ EXPECT_GT(r.num_buffers, 0);
+ EXPECT_EQ(r.num_buffers, 5); // ceil(0.155/0.020)=8, capped by floor(0.100/0.020)=5
+ // Launch hold strictly improved; capture hold not driven below margin.
+ EXPECT_GT(-0.155 + r.delta_realized, -0.155);
+ EXPECT_GE(0.100 - r.delta_realized, 0.0);
+}
+
+// Ample capture room -> push exactly enough to clear the launch violation.
+TEST(MacroSkew, LaunchFixedWhenCaptureRoomAmple)
+{
+ const MacroSkewResult r = computeMacroSkew({.s_cap_hold = 0.400,
+ .s_launch_hold = -0.100,
+ .buffer_delay = 0.020,
+ .hold_margin = 0.0});
+ EXPECT_EQ(r.num_buffers, 5); // ceil(0.100/0.020), capture room (20) not binding
+ EXPECT_GE(-0.100 + r.delta_realized, 0.0); // launch hold now met
+}
+
+// A macro that never launches (no launch path -> +inf) is never pushed later,
+// even when its capture hold is violated (pushing later only worsens capture).
+TEST(MacroSkew, NoLaunchPathNoPush)
+{
+ const MacroSkewResult r = computeMacroSkew({.s_cap_hold = -0.050,
+ .s_launch_hold = kInf,
+ .buffer_delay = 0.020,
+ .hold_margin = 0.0});
+ EXPECT_EQ(r.num_buffers, 0);
+}
+
+// Launch hold already met -> no push.
+TEST(MacroSkew, LaunchOkNoPush)
+{
+ const MacroSkewResult r = computeMacroSkew({.s_cap_hold = 0.20,
+ .s_launch_hold = 0.15,
+ .buffer_delay = 0.020,
+ .hold_margin = 0.0});
+ EXPECT_EQ(r.num_buffers, 0);
+}
+
+// Both hold sides violated: pushing later can't help launch without worsening
+// capture (capture room <= 0) -> no push; it's a data-path / global problem.
+TEST(MacroSkew, BothHoldViolatedNoPush)
+{
+ const MacroSkewResult r = computeMacroSkew({.s_cap_hold = -0.05,
+ .s_launch_hold = -0.10,
+ .buffer_delay = 0.020,
+ .hold_margin = 0.0});
+ EXPECT_EQ(r.num_buffers, 0);
+}
+
+// Tiny capture room caps the push well short of fully clearing a big launch
+// violation (the fix never sacrifices capture hold).
+TEST(MacroSkew, CaptureRoomCaps)
+{
+ const MacroSkewResult r = computeMacroSkew({.s_cap_hold = 0.030,
+ .s_launch_hold = -0.200,
+ .buffer_delay = 0.020,
+ .hold_margin = 0.0});
+ EXPECT_EQ(r.num_buffers, 1); // ceil(0.200/0.020)=10, capped by floor(0.030/0.020)=1
+ EXPECT_GE(0.030 - r.delta_realized, 0.0);
+}
+
+// Sub-buffer push rounds UP so the violation is actually cleared.
+TEST(MacroSkew, QuantizationRoundsUp)
+{
+ const MacroSkewResult r = computeMacroSkew({.s_cap_hold = 1.0,
+ .s_launch_hold = -0.037,
+ .buffer_delay = 0.020,
+ .hold_margin = 0.0});
+ EXPECT_EQ(r.num_buffers, 2); // ceil(0.037/0.020)=ceil(1.85)
+ EXPECT_DOUBLE_EQ(r.delta_realized, 2 * 0.020);
+}
+
+// A launch hold below a positive margin (not yet negative) is still pushed to
+// reach the margin.
+TEST(MacroSkew, PushesToReachHoldMargin)
+{
+ const MacroSkewResult r = computeMacroSkew({.s_cap_hold = 0.50,
+ .s_launch_hold = 0.005,
+ .buffer_delay = 0.020,
+ .hold_margin = 0.010});
+ EXPECT_EQ(r.num_buffers, 1); // ceil((0.010-0.005)/0.020)
+}
+
+} // namespace cts
Additional Context
Known weaknesses of the prototype:
- Greedy per-macro iterate. It is coordinate-descent on the underlying Fishburn LP and converges here because macro↔register coupling dominates and macro↔macro is sparse. The principled version is the one-shot difference-constraint-graph solve (Bellman-Ford feasibility / LP max-min-slack), and further out prescribed-skew DME with the offset as a first-class tree-construction input (Fishburn, IEEE Trans. Computers 1990; Chao/Hsu/Ho, DAC 1992; Cong/Kahng/Koh/Tsao, TODAES 1999; Tsao/Koh, TODAES 2002).
- Quantized, bolt-on realization. A delay-buffer post-pass quantizes to buffer-stage granularity and does not track the macro's internal clock tree across PVT/OCV, so residual mismatch can reappear as corner-sensitive boundary hold. Fine for design-space exploration; a known limitation for tape-out.
- Hold-only, single corner. A full treatment honors
min_clock_tree_path(hold) andmax_clock_tree_path(setup) per corner and minimizes clock reconvergence pessimism at macro boundaries. - Tested on one design family so far; we are still validating on larger configurations of it.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.