[LV][VPlan] Add initial support for CSA vectorization #121222
Conversation
michaelmaitland
requested review from
artagnon,
fhahn,
alexey-bataev,
huntergr-arm,
ayalz,
david-arm and
ElvisWang123
|
@llvm/pr-subscribers-vectorizers @llvm/pr-subscribers-backend-risc-v Author: Michael Maitland (michaelmaitland) ChangesThis patch adds initial support for CSA vectorization LLVM. This new class For example: Using pseudo-LLVM code this can be vectorized as On each iteration, we track whether any lane in the widened condition was active, This transformation works the same way for integer, pointer, and floating point In the vectorization of a CSA, we will be introducing recipes into the vector A detailed explanation of the concept can be found here. This patch is further tested in llvm/llvm-test-suite#155. This patch contains only the non-EVL related code. The is based on the larger Patch is 232.25 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/121222.diff 18 Files Affected:
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index e8041e22b031ce..b085f4bd173afc 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -6,7 +6,8 @@
//
//===----------------------------------------------------------------------===//
//
-// This file "describes" induction and recurrence variables.
+// This file "describes" induction, recurrence, and conditional scalar
+// assignment variables.
//
//===----------------------------------------------------------------------===//
@@ -423,6 +424,71 @@ class InductionDescriptor {
SmallVector<Instruction *, 2> RedundantCasts;
};
+/// A Conditional Scalar Assignment is an assignment from an initial
+/// scalar that may or may not occur.
+class ConditionalScalarAssignmentDescriptor {
+ /// If the conditional assignment occurs inside a loop, then Phi chooses
+ /// the value of the assignment from the entry block or the loop body block.
+ PHINode *Phi = nullptr;
+
+ /// The initial value of the ConditionalScalarAssignment. If the condition
+ /// guarding the assignment is not met, then the assignment retains this
+ /// value.
+ Value *InitScalar = nullptr;
+
+ /// The Instruction that conditionally assigned to inside the loop.
+ Instruction *Assignment = nullptr;
+
+ /// Create a ConditionalScalarAssignmentDescriptor that models a valid
+ /// conditional scalar assignment with its members initialized correctly.
+ ConditionalScalarAssignmentDescriptor(PHINode *Phi, Instruction *Assignment,
+ Value *InitScalar)
+ : Phi(Phi), InitScalar(InitScalar), Assignment(Assignment) {}
+
+public:
+ /// Create a ConditionalScalarAssignmentDescriptor that models an invalid
+ /// ConditionalScalarAssignment.
+ ConditionalScalarAssignmentDescriptor() = default;
+
+ /// If Phi is the root of a ConditionalScalarAssignment, set
+ /// ConditionalScalarAssignmentDesc as the ConditionalScalarAssignment rooted
+ /// by Phi. Otherwise, return a false, leaving ConditionalScalarAssignmentDesc
+ /// unmodified.
+ static bool isConditionalScalarAssignmentPhi(
+ PHINode *Phi, Loop *TheLoop,
+ ConditionalScalarAssignmentDescriptor &Desc);
+
+ operator bool() const { return isValid(); }
+
+ /// Returns whether SI is the Assignment in ConditionalScalarAssignment
+ static bool isConditionalScalarAssignmentSelect(
+ ConditionalScalarAssignmentDescriptor Desc, SelectInst *SI) {
+ return Desc.getAssignment() == SI;
+ }
+
+ /// Return whether this ConditionalScalarAssignmentDescriptor models a valid
+ /// ConditionalScalarAssignment.
+ bool isValid() const { return Phi && InitScalar && Assignment; }
+
+ /// Return the PHI that roots this ConditionalScalarAssignment.
+ PHINode *getPhi() const { return Phi; }
+
+ /// Return the initial value of the ConditionalScalarAssignment. This is the
+ /// value if the conditional assignment does not occur.
+ Value *getInitScalar() const { return InitScalar; }
+
+ /// The Instruction that is used after the loop
+ Instruction *getAssignment() const { return Assignment; }
+
+ /// Return the condition that this ConditionalScalarAssignment is conditional
+ /// upon.
+ Value *getCond() const {
+ if (auto *SI = dyn_cast_or_null<SelectInst>(Assignment))
+ return SI->getCondition();
+ return nullptr;
+ }
+};
+
} // end namespace llvm
#endif // LLVM_ANALYSIS_IVDESCRIPTORS_H
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfo.h b/llvm/include/llvm/Analysis/TargetTransformInfo.h
index c6b846f96f1622..b41dbb6582f76f 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfo.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfo.h
@@ -1852,6 +1852,10 @@ class TargetTransformInfo {
: EVLParamStrategy(EVLParamStrategy), OpStrategy(OpStrategy) {}
};
+ /// \returns true if the loop vectorizer should vectorize conditional
+ /// scalar assignments for the target.
+ bool enableConditionalScalarAssignmentVectorization() const;
+
/// \returns How the target needs this vector-predicated operation to be
/// transformed.
VPLegalization getVPLegalizationStrategy(const VPIntrinsic &PI) const;
@@ -2305,6 +2309,7 @@ class TargetTransformInfo::Concept {
SmallVectorImpl<Use *> &OpsToSink) const = 0;
virtual bool isVectorShiftByScalarCheap(Type *Ty) const = 0;
+ virtual bool enableConditionalScalarAssignmentVectorization() const = 0;
virtual VPLegalization
getVPLegalizationStrategy(const VPIntrinsic &PI) const = 0;
virtual bool hasArmWideBranch(bool Thumb) const = 0;
@@ -3130,6 +3135,10 @@ class TargetTransformInfo::Model final : public TargetTransformInfo::Concept {
return Impl.isVectorShiftByScalarCheap(Ty);
}
+ bool enableConditionalScalarAssignmentVectorization() const override {
+ return Impl.enableConditionalScalarAssignmentVectorization();
+ }
+
VPLegalization
getVPLegalizationStrategy(const VPIntrinsic &PI) const override {
return Impl.getVPLegalizationStrategy(PI);
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
index 5fa0c46ad292d8..0cb081d48991f2 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
@@ -1030,6 +1030,8 @@ class TargetTransformInfoImplBase {
bool isVectorShiftByScalarCheap(Type *Ty) const { return false; }
+ bool enableConditionalScalarAssignmentVectorization() const { return false; }
+
TargetTransformInfo::VPLegalization
getVPLegalizationStrategy(const VPIntrinsic &PI) const {
return TargetTransformInfo::VPLegalization(
diff --git a/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h b/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
index fbe80eddbae07a..48c9077d653278 100644
--- a/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
+++ b/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
@@ -269,6 +269,12 @@ class LoopVectorizationLegality {
/// induction descriptor.
using InductionList = MapVector<PHINode *, InductionDescriptor>;
+ /// ConditionalScalarAssignmentList contains the
+ /// ConditionalScalarAssignmentDescriptors for all the conditional scalar
+ /// assignments that were found in the loop, rooted by their phis.
+ using ConditionalScalarAssignmentList =
+ MapVector<PHINode *, ConditionalScalarAssignmentDescriptor>;
+
/// RecurrenceSet contains the phi nodes that are recurrences other than
/// inductions and reductions.
using RecurrenceSet = SmallPtrSet<const PHINode *, 8>;
@@ -321,6 +327,18 @@ class LoopVectorizationLegality {
/// Returns True if V is a Phi node of an induction variable in this loop.
bool isInductionPhi(const Value *V) const;
+ /// Returns the conditional scalar assignments found in the loop.
+ const ConditionalScalarAssignmentList &
+ getConditionalScalarAssignments() const {
+ return ConditionalScalarAssignments;
+ }
+
+ /// Returns true if Phi is the root of a conditional scalar assignments in the
+ /// loop.
+ bool isConditionalScalarAssignmentPhi(PHINode *Phi) const {
+ return ConditionalScalarAssignments.count(Phi) != 0;
+ }
+
/// Returns a pointer to the induction descriptor, if \p Phi is an integer or
/// floating point induction.
const InductionDescriptor *getIntOrFpInductionDescriptor(PHINode *Phi) const;
@@ -550,6 +568,12 @@ class LoopVectorizationLegality {
void addInductionPhi(PHINode *Phi, const InductionDescriptor &ID,
SmallPtrSetImpl<Value *> &AllowedExit);
+ /// Updates the vetorization state by adding \p Phi to the
+ /// ConditionalScalarAssignment list.
+ void addConditionalScalarAssignmentPhi(
+ PHINode *Phi, const ConditionalScalarAssignmentDescriptor &Desc,
+ SmallPtrSetImpl<Value *> &AllowedExit);
+
/// The loop that we evaluate.
Loop *TheLoop;
@@ -594,6 +618,9 @@ class LoopVectorizationLegality {
/// variables can be pointers.
InductionList Inductions;
+ /// Holds the conditional scalar assignments
+ ConditionalScalarAssignmentList ConditionalScalarAssignments;
+
/// Holds all the casts that participate in the update chain of the induction
/// variables, and that have been proven to be redundant (possibly under a
/// runtime guard). These casts can be ignored when creating the vectorized
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index f74ede4450ce52..259af79f6cdba5 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -6,7 +6,8 @@
//
//===----------------------------------------------------------------------===//
//
-// This file "describes" induction and recurrence variables.
+// This file "describes" induction, recurrence, and conditional scalar
+// assignment variables.
//
//===----------------------------------------------------------------------===//
@@ -1570,3 +1571,60 @@ bool InductionDescriptor::isInductionPHI(
D = InductionDescriptor(StartValue, IK_PtrInduction, Step);
return true;
}
+
+/// Return ConditionalScalarAssignmentDescriptor that describes a
+/// ConditionalScalarAssignment that matches one of these patterns:
+/// phi loop_inv, (select cmp, value, phi)
+/// phi loop_inv, (select cmp, phi, value)
+/// phi (select cmp, value, phi), loop_inv
+/// phi (select cmp, phi, value), loop_inv
+/// If the ConditionalScalarAssignment does not match any of these paterns,
+/// return a ConditionalScalarAssignmentDescriptor that describes an
+/// InvalidConditionalScalarAssignment.
+bool ConditionalScalarAssignmentDescriptor::isConditionalScalarAssignmentPhi(
+ PHINode *Phi, Loop *TheLoop, ConditionalScalarAssignmentDescriptor &Desc) {
+
+ // Must be a scalar.
+ Type *Type = Phi->getType();
+ if (!Type->isIntegerTy() && !Type->isFloatingPointTy() &&
+ !Type->isPointerTy())
+ return false;
+
+ // Match phi loop_inv, (select cmp, value, phi)
+ // or phi loop_inv, (select cmp, phi, value)
+ // or phi (select cmp, value, phi), loop_inv
+ // or phi (select cmp, phi, value), loop_inv
+ if (Phi->getNumIncomingValues() != 2)
+ return false;
+ auto SelectInstIt = find_if(Phi->incoming_values(), [&Phi](const Use &U) {
+ return match(U.get(), m_Select(m_Value(), m_Specific(Phi), m_Value())) ||
+ match(U.get(), m_Select(m_Value(), m_Value(), m_Specific(Phi)));
+ });
+ if (SelectInstIt == Phi->incoming_values().end())
+ return false;
+ auto LoopInvIt = find_if(Phi->incoming_values(), [&](Use &U) {
+ return U.get() != *SelectInstIt && TheLoop->isLoopInvariant(U.get());
+ });
+ if (LoopInvIt == Phi->incoming_values().end())
+ return false;
+
+ // Phi or Sel must be used only outside the loop,
+ // excluding if Phi use Sel or Sel use Phi
+ auto IsOnlyUsedOutsideLoop = [&](Value *V, Value *Ignore) {
+ return all_of(V->users(), [Ignore, TheLoop](User *U) {
+ if (U == Ignore)
+ return true;
+ if (auto *I = dyn_cast<Instruction>(U))
+ return !TheLoop->contains(I);
+ return true;
+ });
+ };
+ Instruction *Select = cast<SelectInst>(SelectInstIt->get());
+ Value *LoopInv = LoopInvIt->get();
+ if (!IsOnlyUsedOutsideLoop(Phi, Select) ||
+ !IsOnlyUsedOutsideLoop(Select, Phi))
+ return false;
+
+ Desc = ConditionalScalarAssignmentDescriptor(Phi, Select, LoopInv);
+ return true;
+}
diff --git a/llvm/lib/Analysis/TargetTransformInfo.cpp b/llvm/lib/Analysis/TargetTransformInfo.cpp
index c62e40db0c5775..2468227be4a0da 100644
--- a/llvm/lib/Analysis/TargetTransformInfo.cpp
+++ b/llvm/lib/Analysis/TargetTransformInfo.cpp
@@ -1373,6 +1373,11 @@ bool TargetTransformInfo::preferEpilogueVectorization() const {
return TTIImpl->preferEpilogueVectorization();
}
+bool TargetTransformInfo::enableConditionalScalarAssignmentVectorization()
+ const {
+ return TTIImpl->enableConditionalScalarAssignmentVectorization();
+}
+
TargetTransformInfo::VPLegalization
TargetTransformInfo::getVPLegalizationStrategy(const VPIntrinsic &VPI) const {
return TTIImpl->getVPLegalizationStrategy(VPI);
diff --git a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
index 49192bd6380223..e469c0225cb0a3 100644
--- a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
+++ b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
@@ -2361,6 +2361,11 @@ bool RISCVTTIImpl::isLegalMaskedExpandLoad(Type *DataTy, Align Alignment) {
return true;
}
+bool RISCVTTIImpl::enableConditionalScalarAssignmentVectorization() const {
+ return ST->hasVInstructions() &&
+ ST->getProcFamily() == RISCVSubtarget::SiFive7;
+}
+
bool RISCVTTIImpl::isLegalMaskedCompressStore(Type *DataTy, Align Alignment) {
auto *VTy = dyn_cast<VectorType>(DataTy);
if (!VTy || VTy->isScalableTy())
diff --git a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h
index bd90bfed6e2c95..9e1b2cb3f3043f 100644
--- a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h
+++ b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h
@@ -306,6 +306,10 @@ class RISCVTTIImpl : public BasicTTIImplBase<RISCVTTIImpl> {
return TLI->isVScaleKnownToBeAPowerOfTwo();
}
+ /// \returns true if the loop vectorizer should vectorize conditional
+ /// scalar assignments for the target.
+ bool enableConditionalScalarAssignmentVectorization() const;
+
/// \returns How the target needs this vector-predicated operation to be
/// transformed.
TargetTransformInfo::VPLegalization
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
index cb0b4641b6492b..2835d9f385ac5f 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
@@ -83,6 +83,10 @@ static cl::opt<bool> EnableHistogramVectorization(
"enable-histogram-loop-vectorization", cl::init(false), cl::Hidden,
cl::desc("Enables autovectorization of some loops containing histograms"));
+static cl::opt<bool> EnableConditionalScalarAssignment(
+ "enable-csa-vectorization", cl::init(false), cl::Hidden,
+ cl::desc("Control whether loop vectorization is enabled"));
+
/// Maximum vectorization interleave count.
static const unsigned MaxInterleaveFactor = 16;
@@ -749,6 +753,18 @@ bool LoopVectorizationLegality::setupOuterLoopInductions() {
return llvm::all_of(Header->phis(), IsSupportedPhi);
}
+void LoopVectorizationLegality::addConditionalScalarAssignmentPhi(
+ PHINode *Phi, const ConditionalScalarAssignmentDescriptor &Desc,
+ SmallPtrSetImpl<Value *> &AllowedExit) {
+ assert(Desc.isValid() &&
+ "Expected Valid ConditionalScalarAssignmentDescriptor");
+ LLVM_DEBUG(
+ dbgs() << "LV: found legal conditional scalar assignment opportunity"
+ << *Phi << "\n");
+ AllowedExit.insert(Phi);
+ ConditionalScalarAssignments.insert({Phi, Desc});
+}
+
/// Checks if a function is scalarizable according to the TLI, in
/// the sense that it should be vectorized and then expanded in
/// multiple scalar calls. This is represented in the
@@ -866,14 +882,27 @@ bool LoopVectorizationLegality::canVectorizeInstrs() {
continue;
}
- // As a last resort, coerce the PHI to a AddRec expression
- // and re-try classifying it a an induction PHI.
+ // Try to coerce the PHI to a AddRec expression and re-try classifying
+ // it a an induction PHI.
if (InductionDescriptor::isInductionPHI(Phi, TheLoop, PSE, ID, true) &&
!IsDisallowedStridedPointerInduction(ID)) {
addInductionPhi(Phi, ID, AllowedExit);
continue;
}
+ // Check if the PHI can be classified as a conditional scalar assignment
+ // PHI.
+ if (EnableConditionalScalarAssignment ||
+ (TTI->enableConditionalScalarAssignmentVectorization() &&
+ EnableConditionalScalarAssignment.getNumOccurrences() == 0)) {
+ ConditionalScalarAssignmentDescriptor Desc;
+ if (ConditionalScalarAssignmentDescriptor::
+ isConditionalScalarAssignmentPhi(Phi, TheLoop, Desc)) {
+ addConditionalScalarAssignmentPhi(Phi, Desc, AllowedExit);
+ continue;
+ }
+ }
+
reportVectorizationFailure("Found an unidentified PHI",
"value that could not be identified as "
"reduction is used outside the loop",
@@ -1844,11 +1873,15 @@ bool LoopVectorizationLegality::canFoldTailByMasking() const {
for (const auto &Reduction : getReductionVars())
ReductionLiveOuts.insert(Reduction.second.getLoopExitInstr());
+ SmallPtrSet<const Value *, 8> CSALiveOuts;
+ for (const auto &CSA : getConditionalScalarAssignments())
+ CSALiveOuts.insert(CSA.second.getAssignment());
+
// TODO: handle non-reduction outside users when tail is folded by masking.
for (auto *AE : AllowedExit) {
// Check that all users of allowed exit values are inside the loop or
- // are the live-out of a reduction.
- if (ReductionLiveOuts.count(AE))
+ // are the live-out of a reduction or conditional scalar assignment.
+ if (ReductionLiveOuts.count(AE) || CSALiveOuts.count(AE))
continue;
for (User *U : AE->users()) {
Instruction *UI = cast<Instruction>(U);
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
index 650a4859780da2..6eba380ceb7a12 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
@@ -174,8 +174,8 @@ class VPBuilder {
new VPInstruction(Opcode, Operands, WrapFlags, DL, Name));
}
- VPValue *createNot(VPValue *Operand, DebugLoc DL = {},
- const Twine &Name = "") {
+ VPInstruction *createNot(VPValue *Operand, DebugLoc DL = {},
+ const Twine &Name = "") {
return createInstruction(VPInstruction::Not, {Operand}, DL, Name);
}
@@ -257,6 +257,26 @@ class VPBuilder {
FPBinOp ? FPBinOp->getFastMathFlags() : FastMathFlags()));
}
+ VPInstruction *createConditionalScalarAssignmentMaskPhi(VPValue *InitMask,
+ DebugLoc DL,
+ const Twine &Name) {
+ return createInstruction(VPInstruction::ConditionalScalarAssignmentMaskPhi,
+ {InitMask}, DL, Name);
+ }
+
+ VPInstruction *createAnyOf(VPValue *Cond, DebugLoc DL, const Twine &Name) {
+ return createInstruction(VPInstruction::AnyOf, {Cond}, DL, Name);
+ }
+
+ VPInstruction *createConditionalScalarAssignmentMaskSel(VPValue *Cond,
+ VPValue *MaskPhi,
+ VPValue *AnyOf,
+ DebugLoc DL,
+ const Twine &Name) {
+ return createInstruction(VPInstruction::ConditionalScalarAssignmentMaskSel,
+ {Cond, MaskPhi, AnyOf}, DL, Name);
+ }
+
//===--------------------------------------------------------------------===//
// RAII helpers.
//===--------------------------------------------------------------------===//
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 355ff40ce770e7..7102e0437caf5c 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -174,6 +174,8 @@ const char LLVMLoopVectorizeFollowupEpilogue[] =
STATISTIC(LoopsVectorized, "Number of loops vectorized");
STATISTIC(LoopsAnalyzed, "Number of loops analyzed for vectorization");
STATISTIC(LoopsEpilogueVectorized, "Number of epilogues vectorized");
+STATISTIC(ConditionalScalarAssignmentsVectorized,
+ "Number of conditional scalar assignments vectorized");
static cl::opt<bool> EnableEpilogueVectorization(
"enable-epilogue-vectorization", cl::init(true), cl::Hidden,
@@ -4635,6 +4637,9 @@ static bool willGenerateVect...
[truncated]
|
|
@llvm/pr-subscribers-llvm-transforms Author: Michael Maitland (michaelmaitland) ChangesThis patch adds initial support for CSA vectorization LLVM. This new class For example: Using pseudo-LLVM code this can be vectorized as On each iteration, we track whether any lane in the widened condition was active, This transformation works the same way for integer, pointer, and floating point In the vectorization of a CSA, we will be introducing recipes into the vector A detailed explanation of the concept can be found here. This patch is further tested in llvm/llvm-test-suite#155. This patch contains only the non-EVL related code. The is based on the larger Patch is 232.25 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/121222.diff 18 Files Affected:
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index e8041e22b031ce..b085f4bd173afc 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -6,7 +6,8 @@
//
//===----------------------------------------------------------------------===//
//
-// This file "describes" induction and recurrence variables.
+// This file "describes" induction, recurrence, and conditional scalar
+// assignment variables.
//
//===----------------------------------------------------------------------===//
@@ -423,6 +424,71 @@ class InductionDescriptor {
SmallVector<Instruction *, 2> RedundantCasts;
};
+/// A Conditional Scalar Assignment is an assignment from an initial
+/// scalar that may or may not occur.
+class ConditionalScalarAssignmentDescriptor {
+ /// If the conditional assignment occurs inside a loop, then Phi chooses
+ /// the value of the assignment from the entry block or the loop body block.
+ PHINode *Phi = nullptr;
+
+ /// The initial value of the ConditionalScalarAssignment. If the condition
+ /// guarding the assignment is not met, then the assignment retains this
+ /// value.
+ Value *InitScalar = nullptr;
+
+ /// The Instruction that conditionally assigned to inside the loop.
+ Instruction *Assignment = nullptr;
+
+ /// Create a ConditionalScalarAssignmentDescriptor that models a valid
+ /// conditional scalar assignment with its members initialized correctly.
+ ConditionalScalarAssignmentDescriptor(PHINode *Phi, Instruction *Assignment,
+ Value *InitScalar)
+ : Phi(Phi), InitScalar(InitScalar), Assignment(Assignment) {}
+
+public:
+ /// Create a ConditionalScalarAssignmentDescriptor that models an invalid
+ /// ConditionalScalarAssignment.
+ ConditionalScalarAssignmentDescriptor() = default;
+
+ /// If Phi is the root of a ConditionalScalarAssignment, set
+ /// ConditionalScalarAssignmentDesc as the ConditionalScalarAssignment rooted
+ /// by Phi. Otherwise, return a false, leaving ConditionalScalarAssignmentDesc
+ /// unmodified.
+ static bool isConditionalScalarAssignmentPhi(
+ PHINode *Phi, Loop *TheLoop,
+ ConditionalScalarAssignmentDescriptor &Desc);
+
+ operator bool() const { return isValid(); }
+
+ /// Returns whether SI is the Assignment in ConditionalScalarAssignment
+ static bool isConditionalScalarAssignmentSelect(
+ ConditionalScalarAssignmentDescriptor Desc, SelectInst *SI) {
+ return Desc.getAssignment() == SI;
+ }
+
+ /// Return whether this ConditionalScalarAssignmentDescriptor models a valid
+ /// ConditionalScalarAssignment.
+ bool isValid() const { return Phi && InitScalar && Assignment; }
+
+ /// Return the PHI that roots this ConditionalScalarAssignment.
+ PHINode *getPhi() const { return Phi; }
+
+ /// Return the initial value of the ConditionalScalarAssignment. This is the
+ /// value if the conditional assignment does not occur.
+ Value *getInitScalar() const { return InitScalar; }
+
+ /// The Instruction that is used after the loop
+ Instruction *getAssignment() const { return Assignment; }
+
+ /// Return the condition that this ConditionalScalarAssignment is conditional
+ /// upon.
+ Value *getCond() const {
+ if (auto *SI = dyn_cast_or_null<SelectInst>(Assignment))
+ return SI->getCondition();
+ return nullptr;
+ }
+};
+
} // end namespace llvm
#endif // LLVM_ANALYSIS_IVDESCRIPTORS_H
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfo.h b/llvm/include/llvm/Analysis/TargetTransformInfo.h
index c6b846f96f1622..b41dbb6582f76f 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfo.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfo.h
@@ -1852,6 +1852,10 @@ class TargetTransformInfo {
: EVLParamStrategy(EVLParamStrategy), OpStrategy(OpStrategy) {}
};
+ /// \returns true if the loop vectorizer should vectorize conditional
+ /// scalar assignments for the target.
+ bool enableConditionalScalarAssignmentVectorization() const;
+
/// \returns How the target needs this vector-predicated operation to be
/// transformed.
VPLegalization getVPLegalizationStrategy(const VPIntrinsic &PI) const;
@@ -2305,6 +2309,7 @@ class TargetTransformInfo::Concept {
SmallVectorImpl<Use *> &OpsToSink) const = 0;
virtual bool isVectorShiftByScalarCheap(Type *Ty) const = 0;
+ virtual bool enableConditionalScalarAssignmentVectorization() const = 0;
virtual VPLegalization
getVPLegalizationStrategy(const VPIntrinsic &PI) const = 0;
virtual bool hasArmWideBranch(bool Thumb) const = 0;
@@ -3130,6 +3135,10 @@ class TargetTransformInfo::Model final : public TargetTransformInfo::Concept {
return Impl.isVectorShiftByScalarCheap(Ty);
}
+ bool enableConditionalScalarAssignmentVectorization() const override {
+ return Impl.enableConditionalScalarAssignmentVectorization();
+ }
+
VPLegalization
getVPLegalizationStrategy(const VPIntrinsic &PI) const override {
return Impl.getVPLegalizationStrategy(PI);
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
index 5fa0c46ad292d8..0cb081d48991f2 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
@@ -1030,6 +1030,8 @@ class TargetTransformInfoImplBase {
bool isVectorShiftByScalarCheap(Type *Ty) const { return false; }
+ bool enableConditionalScalarAssignmentVectorization() const { return false; }
+
TargetTransformInfo::VPLegalization
getVPLegalizationStrategy(const VPIntrinsic &PI) const {
return TargetTransformInfo::VPLegalization(
diff --git a/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h b/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
index fbe80eddbae07a..48c9077d653278 100644
--- a/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
+++ b/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
@@ -269,6 +269,12 @@ class LoopVectorizationLegality {
/// induction descriptor.
using InductionList = MapVector<PHINode *, InductionDescriptor>;
+ /// ConditionalScalarAssignmentList contains the
+ /// ConditionalScalarAssignmentDescriptors for all the conditional scalar
+ /// assignments that were found in the loop, rooted by their phis.
+ using ConditionalScalarAssignmentList =
+ MapVector<PHINode *, ConditionalScalarAssignmentDescriptor>;
+
/// RecurrenceSet contains the phi nodes that are recurrences other than
/// inductions and reductions.
using RecurrenceSet = SmallPtrSet<const PHINode *, 8>;
@@ -321,6 +327,18 @@ class LoopVectorizationLegality {
/// Returns True if V is a Phi node of an induction variable in this loop.
bool isInductionPhi(const Value *V) const;
+ /// Returns the conditional scalar assignments found in the loop.
+ const ConditionalScalarAssignmentList &
+ getConditionalScalarAssignments() const {
+ return ConditionalScalarAssignments;
+ }
+
+ /// Returns true if Phi is the root of a conditional scalar assignments in the
+ /// loop.
+ bool isConditionalScalarAssignmentPhi(PHINode *Phi) const {
+ return ConditionalScalarAssignments.count(Phi) != 0;
+ }
+
/// Returns a pointer to the induction descriptor, if \p Phi is an integer or
/// floating point induction.
const InductionDescriptor *getIntOrFpInductionDescriptor(PHINode *Phi) const;
@@ -550,6 +568,12 @@ class LoopVectorizationLegality {
void addInductionPhi(PHINode *Phi, const InductionDescriptor &ID,
SmallPtrSetImpl<Value *> &AllowedExit);
+ /// Updates the vetorization state by adding \p Phi to the
+ /// ConditionalScalarAssignment list.
+ void addConditionalScalarAssignmentPhi(
+ PHINode *Phi, const ConditionalScalarAssignmentDescriptor &Desc,
+ SmallPtrSetImpl<Value *> &AllowedExit);
+
/// The loop that we evaluate.
Loop *TheLoop;
@@ -594,6 +618,9 @@ class LoopVectorizationLegality {
/// variables can be pointers.
InductionList Inductions;
+ /// Holds the conditional scalar assignments
+ ConditionalScalarAssignmentList ConditionalScalarAssignments;
+
/// Holds all the casts that participate in the update chain of the induction
/// variables, and that have been proven to be redundant (possibly under a
/// runtime guard). These casts can be ignored when creating the vectorized
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index f74ede4450ce52..259af79f6cdba5 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -6,7 +6,8 @@
//
//===----------------------------------------------------------------------===//
//
-// This file "describes" induction and recurrence variables.
+// This file "describes" induction, recurrence, and conditional scalar
+// assignment variables.
//
//===----------------------------------------------------------------------===//
@@ -1570,3 +1571,60 @@ bool InductionDescriptor::isInductionPHI(
D = InductionDescriptor(StartValue, IK_PtrInduction, Step);
return true;
}
+
+/// Return ConditionalScalarAssignmentDescriptor that describes a
+/// ConditionalScalarAssignment that matches one of these patterns:
+/// phi loop_inv, (select cmp, value, phi)
+/// phi loop_inv, (select cmp, phi, value)
+/// phi (select cmp, value, phi), loop_inv
+/// phi (select cmp, phi, value), loop_inv
+/// If the ConditionalScalarAssignment does not match any of these paterns,
+/// return a ConditionalScalarAssignmentDescriptor that describes an
+/// InvalidConditionalScalarAssignment.
+bool ConditionalScalarAssignmentDescriptor::isConditionalScalarAssignmentPhi(
+ PHINode *Phi, Loop *TheLoop, ConditionalScalarAssignmentDescriptor &Desc) {
+
+ // Must be a scalar.
+ Type *Type = Phi->getType();
+ if (!Type->isIntegerTy() && !Type->isFloatingPointTy() &&
+ !Type->isPointerTy())
+ return false;
+
+ // Match phi loop_inv, (select cmp, value, phi)
+ // or phi loop_inv, (select cmp, phi, value)
+ // or phi (select cmp, value, phi), loop_inv
+ // or phi (select cmp, phi, value), loop_inv
+ if (Phi->getNumIncomingValues() != 2)
+ return false;
+ auto SelectInstIt = find_if(Phi->incoming_values(), [&Phi](const Use &U) {
+ return match(U.get(), m_Select(m_Value(), m_Specific(Phi), m_Value())) ||
+ match(U.get(), m_Select(m_Value(), m_Value(), m_Specific(Phi)));
+ });
+ if (SelectInstIt == Phi->incoming_values().end())
+ return false;
+ auto LoopInvIt = find_if(Phi->incoming_values(), [&](Use &U) {
+ return U.get() != *SelectInstIt && TheLoop->isLoopInvariant(U.get());
+ });
+ if (LoopInvIt == Phi->incoming_values().end())
+ return false;
+
+ // Phi or Sel must be used only outside the loop,
+ // excluding if Phi use Sel or Sel use Phi
+ auto IsOnlyUsedOutsideLoop = [&](Value *V, Value *Ignore) {
+ return all_of(V->users(), [Ignore, TheLoop](User *U) {
+ if (U == Ignore)
+ return true;
+ if (auto *I = dyn_cast<Instruction>(U))
+ return !TheLoop->contains(I);
+ return true;
+ });
+ };
+ Instruction *Select = cast<SelectInst>(SelectInstIt->get());
+ Value *LoopInv = LoopInvIt->get();
+ if (!IsOnlyUsedOutsideLoop(Phi, Select) ||
+ !IsOnlyUsedOutsideLoop(Select, Phi))
+ return false;
+
+ Desc = ConditionalScalarAssignmentDescriptor(Phi, Select, LoopInv);
+ return true;
+}
diff --git a/llvm/lib/Analysis/TargetTransformInfo.cpp b/llvm/lib/Analysis/TargetTransformInfo.cpp
index c62e40db0c5775..2468227be4a0da 100644
--- a/llvm/lib/Analysis/TargetTransformInfo.cpp
+++ b/llvm/lib/Analysis/TargetTransformInfo.cpp
@@ -1373,6 +1373,11 @@ bool TargetTransformInfo::preferEpilogueVectorization() const {
return TTIImpl->preferEpilogueVectorization();
}
+bool TargetTransformInfo::enableConditionalScalarAssignmentVectorization()
+ const {
+ return TTIImpl->enableConditionalScalarAssignmentVectorization();
+}
+
TargetTransformInfo::VPLegalization
TargetTransformInfo::getVPLegalizationStrategy(const VPIntrinsic &VPI) const {
return TTIImpl->getVPLegalizationStrategy(VPI);
diff --git a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
index 49192bd6380223..e469c0225cb0a3 100644
--- a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
+++ b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
@@ -2361,6 +2361,11 @@ bool RISCVTTIImpl::isLegalMaskedExpandLoad(Type *DataTy, Align Alignment) {
return true;
}
+bool RISCVTTIImpl::enableConditionalScalarAssignmentVectorization() const {
+ return ST->hasVInstructions() &&
+ ST->getProcFamily() == RISCVSubtarget::SiFive7;
+}
+
bool RISCVTTIImpl::isLegalMaskedCompressStore(Type *DataTy, Align Alignment) {
auto *VTy = dyn_cast<VectorType>(DataTy);
if (!VTy || VTy->isScalableTy())
diff --git a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h
index bd90bfed6e2c95..9e1b2cb3f3043f 100644
--- a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h
+++ b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h
@@ -306,6 +306,10 @@ class RISCVTTIImpl : public BasicTTIImplBase<RISCVTTIImpl> {
return TLI->isVScaleKnownToBeAPowerOfTwo();
}
+ /// \returns true if the loop vectorizer should vectorize conditional
+ /// scalar assignments for the target.
+ bool enableConditionalScalarAssignmentVectorization() const;
+
/// \returns How the target needs this vector-predicated operation to be
/// transformed.
TargetTransformInfo::VPLegalization
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
index cb0b4641b6492b..2835d9f385ac5f 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
@@ -83,6 +83,10 @@ static cl::opt<bool> EnableHistogramVectorization(
"enable-histogram-loop-vectorization", cl::init(false), cl::Hidden,
cl::desc("Enables autovectorization of some loops containing histograms"));
+static cl::opt<bool> EnableConditionalScalarAssignment(
+ "enable-csa-vectorization", cl::init(false), cl::Hidden,
+ cl::desc("Control whether loop vectorization is enabled"));
+
/// Maximum vectorization interleave count.
static const unsigned MaxInterleaveFactor = 16;
@@ -749,6 +753,18 @@ bool LoopVectorizationLegality::setupOuterLoopInductions() {
return llvm::all_of(Header->phis(), IsSupportedPhi);
}
+void LoopVectorizationLegality::addConditionalScalarAssignmentPhi(
+ PHINode *Phi, const ConditionalScalarAssignmentDescriptor &Desc,
+ SmallPtrSetImpl<Value *> &AllowedExit) {
+ assert(Desc.isValid() &&
+ "Expected Valid ConditionalScalarAssignmentDescriptor");
+ LLVM_DEBUG(
+ dbgs() << "LV: found legal conditional scalar assignment opportunity"
+ << *Phi << "\n");
+ AllowedExit.insert(Phi);
+ ConditionalScalarAssignments.insert({Phi, Desc});
+}
+
/// Checks if a function is scalarizable according to the TLI, in
/// the sense that it should be vectorized and then expanded in
/// multiple scalar calls. This is represented in the
@@ -866,14 +882,27 @@ bool LoopVectorizationLegality::canVectorizeInstrs() {
continue;
}
- // As a last resort, coerce the PHI to a AddRec expression
- // and re-try classifying it a an induction PHI.
+ // Try to coerce the PHI to a AddRec expression and re-try classifying
+ // it a an induction PHI.
if (InductionDescriptor::isInductionPHI(Phi, TheLoop, PSE, ID, true) &&
!IsDisallowedStridedPointerInduction(ID)) {
addInductionPhi(Phi, ID, AllowedExit);
continue;
}
+ // Check if the PHI can be classified as a conditional scalar assignment
+ // PHI.
+ if (EnableConditionalScalarAssignment ||
+ (TTI->enableConditionalScalarAssignmentVectorization() &&
+ EnableConditionalScalarAssignment.getNumOccurrences() == 0)) {
+ ConditionalScalarAssignmentDescriptor Desc;
+ if (ConditionalScalarAssignmentDescriptor::
+ isConditionalScalarAssignmentPhi(Phi, TheLoop, Desc)) {
+ addConditionalScalarAssignmentPhi(Phi, Desc, AllowedExit);
+ continue;
+ }
+ }
+
reportVectorizationFailure("Found an unidentified PHI",
"value that could not be identified as "
"reduction is used outside the loop",
@@ -1844,11 +1873,15 @@ bool LoopVectorizationLegality::canFoldTailByMasking() const {
for (const auto &Reduction : getReductionVars())
ReductionLiveOuts.insert(Reduction.second.getLoopExitInstr());
+ SmallPtrSet<const Value *, 8> CSALiveOuts;
+ for (const auto &CSA : getConditionalScalarAssignments())
+ CSALiveOuts.insert(CSA.second.getAssignment());
+
// TODO: handle non-reduction outside users when tail is folded by masking.
for (auto *AE : AllowedExit) {
// Check that all users of allowed exit values are inside the loop or
- // are the live-out of a reduction.
- if (ReductionLiveOuts.count(AE))
+ // are the live-out of a reduction or conditional scalar assignment.
+ if (ReductionLiveOuts.count(AE) || CSALiveOuts.count(AE))
continue;
for (User *U : AE->users()) {
Instruction *UI = cast<Instruction>(U);
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
index 650a4859780da2..6eba380ceb7a12 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
@@ -174,8 +174,8 @@ class VPBuilder {
new VPInstruction(Opcode, Operands, WrapFlags, DL, Name));
}
- VPValue *createNot(VPValue *Operand, DebugLoc DL = {},
- const Twine &Name = "") {
+ VPInstruction *createNot(VPValue *Operand, DebugLoc DL = {},
+ const Twine &Name = "") {
return createInstruction(VPInstruction::Not, {Operand}, DL, Name);
}
@@ -257,6 +257,26 @@ class VPBuilder {
FPBinOp ? FPBinOp->getFastMathFlags() : FastMathFlags()));
}
+ VPInstruction *createConditionalScalarAssignmentMaskPhi(VPValue *InitMask,
+ DebugLoc DL,
+ const Twine &Name) {
+ return createInstruction(VPInstruction::ConditionalScalarAssignmentMaskPhi,
+ {InitMask}, DL, Name);
+ }
+
+ VPInstruction *createAnyOf(VPValue *Cond, DebugLoc DL, const Twine &Name) {
+ return createInstruction(VPInstruction::AnyOf, {Cond}, DL, Name);
+ }
+
+ VPInstruction *createConditionalScalarAssignmentMaskSel(VPValue *Cond,
+ VPValue *MaskPhi,
+ VPValue *AnyOf,
+ DebugLoc DL,
+ const Twine &Name) {
+ return createInstruction(VPInstruction::ConditionalScalarAssignmentMaskSel,
+ {Cond, MaskPhi, AnyOf}, DL, Name);
+ }
+
//===--------------------------------------------------------------------===//
// RAII helpers.
//===--------------------------------------------------------------------===//
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 355ff40ce770e7..7102e0437caf5c 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -174,6 +174,8 @@ const char LLVMLoopVectorizeFollowupEpilogue[] =
STATISTIC(LoopsVectorized, "Number of loops vectorized");
STATISTIC(LoopsAnalyzed, "Number of loops analyzed for vectorization");
STATISTIC(LoopsEpilogueVectorized, "Number of epilogues vectorized");
+STATISTIC(ConditionalScalarAssignmentsVectorized,
+ "Number of conditional scalar assignments vectorized");
static cl::opt<bool> EnableEpilogueVectorization(
"enable-epilogue-vectorization", cl::init(true), cl::Hidden,
@@ -4635,6 +4637,9 @@ static bool willGenerateVect...
[truncated]
|
|
✅ With the latest revision this PR passed the C/C++ code formatter. |
michaelmaitland
force-pushed
the
csa-noevl
branch
from
8bc0731 to
b2ac876
Compare
| // preheader and middle block. It also contains recipes that are not backed by | ||
| // underlying instructions in the original loop. This makes it difficult to | ||
| // model in the legacy cost model. | ||
| if (!Legal.getConditionalScalarAssignments().empty()) |
There was a problem hiding this comment.
You said:
Better check for the CSA recipes?
What is the reason to walk all recipes in the plan looking for CSA recipes when we can do this check in O(1) like this?
| @@ -423,6 +424,71 @@ class InductionDescriptor { | |||
| SmallVector<Instruction *, 2> RedundantCasts; | |||
| }; | |||
|
|
|||
| /// A Conditional Scalar Assignment is an assignment from an initial | |||
| /// scalar that may or may not occur. | |||
| class ConditionalScalarAssignmentDescriptor { | |||
There was a problem hiding this comment.
You said:
I don't think CSA is a very common term, would be good to have a more descriptive name if possible
Intel has used the term conditional scalar assingmnet. I have abbreviated it as CSA for short. I have documented the acronym in the code in this patch in multiple places
/// A Conditional Scalar Assignment (CSA) is an assignment from an initial
/// scalar that may or may not occur.
// This file "describes" induction, recurrence, and conditional scalar
// assignment (CSA) variables.
STATISTIC(CSAsVectorized,
"Number of conditional scalar assignments vectorized");
I thought that ConditionalScalarAssignmentDescriptor, createConditionalScalarAssignmentMaskPhi, and VPConditionalScalarAssignmentDescriptorExtractScalarRecipe were quite long for example.
Do you have any suggestion on what you'd like it to be named? Is expanding CSA to ConditionalScalarAssignment everywhere your preference?
For now, I've tried to be proactive and did some renaming as a fixup in this patch. Please let me know what you think.
| @@ -0,0 +1,68 @@ | |||
| ; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 5 | |||
There was a problem hiding this comment.
@fhahn, you said:
I think I am missing something, but the non-EVL codegen doens't seem to use and predicated memory accesses, so it should be able to do it with a generic target?
Yes, the non-evl codegen can do it with generic target, but the EVL related RUN line requires us to be target specific. But then we'd have two test files with the same test cases, only differing by RUN lines. I can make that change if you'd prefer. It was my opinion that it is better to not duplicate the test cases and compare differences in vectorization in one file, but if you disagree, I am content to concede.
|
ping |
michaelmaitland
force-pushed
the
csa-noevl
branch
2 times, most recently
from
63a40e2 to
95bc608
Compare
| br i1 %exitcond.not, label %exit, label %loop | ||
| } | ||
|
|
||
| ; CHECK: VPlan 'Initial VPlan for VF={vscale x 1},UF>=1' { |
There was a problem hiding this comment.
How did the auto-generated CHECK lines end up after the function? I think the VPlan-based testing can be made more extensive.
There was a problem hiding this comment.
Sorry, this was not actually auto generated. I updated the file to remove the first line. Would you like to see any other CHECK lines in this file? If so, what would you like to see specifically?
There was a problem hiding this comment.
Is it possible to also include an EVL test? I think VPlan-based testing can be made more useful by also forcing different VFs? I think we can also report instruction costs, since the cost-computation in the code is quite non-trivial.
There was a problem hiding this comment.
Is it possible to also include an EVL test?
Done
I think VPlan-based testing can be made more useful by also forcing different VFs?
This seems redundant to me. It should not matter much for the VPlan.
I think we can also report instruction costs, since the cost-computation in the code is quite non-trivial.
I'll add another test here for this. I think it could be a good idea to have this depend on forced VFs.
michaelmaitland
force-pushed
the
csa-noevl
branch
2 times, most recently
from
22fa83d to
49c8989
Compare
| br i1 %exitcond.not, label %exit, label %loop | ||
| } | ||
|
|
||
| ; CHECK: VPlan 'Initial VPlan for VF={vscale x 1},UF>=1' { |
There was a problem hiding this comment.
Is it possible to also include an EVL test? I think VPlan-based testing can be made more useful by also forcing different VFs? I think we can also report instruction costs, since the cost-computation in the code is quite non-trivial.
llvm/test/Transforms/LoopVectorize/RISCV/conditional-scalar-assignment.ll
Outdated
Show resolved
Hide resolved
|
ping! |
|
The motivating example of suggests to vectorize this as arguing that it's better to pass vmask as a live-out and sink looking for its last turned-on lane to after the loop, instead of looking for it inside the loop and passing i as live-out. Continuing with this argument, better to also sink the loading of a[i] to after the loop, instead of loading vectorized va with mask inside the loop? In general, there may be some function t=f(i) of i that produces the value t being conditionally overwritten, e.g., think of f(i) as a polynomial of i. Such a function f should arguably be sunk and computed once after the loop - based on figuring out the (at most one) relevant iteration i. The reduction becomes a "FindLast" reduction once this function is sunk. Sound reasonable? This is reminiscent of sinking the "x, y" of AnyOf to produce a boolean reduction followed by an "anyof ? x : y" function, rather than carrying x and y inside the loop, see @fhahn's commit bccb7ed. Finally, note that LV already supports some sort of "CSA", introduced by @annamthomas some years ago in https://reviews.llvm.org/D52656; see test variant_val_store_to_inv_address_conditional in llvm/test/Transforms/LoopVectorize/X86/invariant-store-vectorization.ll - a conditional store to the same address could be converted into a conditional scalar assignment coupled with sinking the conditional store to after the loop. |
|
@ayalz thanks for the review!
I agree in the example you bring up in your example. But consider a case where a[i] is needed unconditionally in the loop. For example if I think there is some room to do such an optimization, but I prefer to leave this as future work. WDYT?
According to #106560 (comment), there is no plan to support
This sounds like a good idea for future improvement of these loops. |
|
Ping |
|
ping! |
This patch adds initial support for CSA vectorization LLVM. This new class
can be characterized by vectorization of assignment to a scalar in a loop,
such that the assignment is conditional from the perspective of its use.
An assignment is conditional in a loop if a value may or may not be assigned
in the loop body.
For example:
```
int t = init_val;
for (int i = 0; i < N; i++) {
if (cond[i])
t = a[i];
}
s = t; // use t
```
Using pseudo-LLVM code this can be vectorized as
```
vector.ph:
...
%t = %init_val
%init.mask = <all-false-vec>
%init.data = <poison-vec> ; uninitialized
vector.body:
...
%mask.phi = phi [%init.mask, %vector.ph], [%new.mask, %vector.body]
%data.phi = phi [%data.mask, %vector.ph], [%new.mask, %vector.body]
%cond.vec = <widened-cmp> ...
%a.vec = <widened-load> %a, %i
%b = <any-lane-active> %cond.vec
%new.mask = select %b, %cond.vec, %mask.phi
%new.data = select %b, %a.vec, %data.phi
...
middle.block:
%s = <extract-last-active-lane> %new.mask, %new.data
```
On each iteration, we track whether any lane in the widened condition was active,
and if it was take the current mask and data as the new mask and data vector.
Then at the end of the loop, the scalar can be extracted only once.
This transformation works the same way for integer, pointer, and floating point
conditional assignment, since the transformation does not require inspection
of the data being assigned.
In the vectorization of a CSA, we will be introducing recipes into the vector
preheader, the vector body, and the middle block. Recipes that are introduced
into the preheader and middle block are executed only one time, and recipes
that are in the vector body will be possibly executed multiple times. The more
times that the vector body is executed, the less of an impact the preheader
and middle block cost have on the overall cost of a CSA.
A detailed explanation of the concept can be found [here](https://discourse.llvm.org/t/vectorization-of-conditional-scalar-assignment-csa/80964).
This patch is further tested in llvm/llvm-test-suite#155.
This patch contains only the non-EVL related code. The is based on the larger
patch of llvm#106560, which contained both EVL and non-EVL related parts.
michaelmaitland
force-pushed
the
csa-noevl
branch
from
9007531 to
7fc4ecd
Compare
There was a problem hiding this comment.
@ayalz thanks for the review!
Continuing with this argument, better to also sink the loading of a[i] to after the loop, instead of loading vectorized va with mask inside the loop?
I agree in the example you bring up in your example. But consider a case where a[i] is needed unconditionally in the loop. For example if
cond[i]is replaced witha[i]. Then doing it after the loop is doing duplicate loads. Additionally, consider thatt = f(a, b, c), then we need to keep track ofa, b, clive out of the loop, which may be tricky.I think there is some room to do such an optimization, but I prefer to leave this as future work. WDYT?
The reduction becomes a "FindLast" reduction once this function is sunk. Sound reasonable?
IIUC the current patch will always load a wide value on each vector iteration vs only doing an extra scalar load at most outside the loop, which seems like it would be more profitable in general?
@ayalz suggestion may help to avoid additional recipes by re-using the existing recipes (and example is sinking stores for reductions to an invariant address).
I added some comments inline to clarify some of the added recieps in the patch, perhaps some of them can be replaced by existing recipes to reduce the complexity.
Recipes also should avoid holding references to underlying IR if possible or modifying existing IR.
According to #106560 (comment), there is no plan to support
FindLastat the moment. Above I describe a scenario where it is not beneficial to do this sinking, so we would not be able to rely onFindLastin all instances.
Now that @Mel-Chen's recent changes landed, I'd hope that adding support for FindLast may be slightly easier, but I am not sure. Maybe @Mel-Chen has additional thoughts?
| cast<PHINode>(getUnderlyingInstr()), *getOperand(0)); | ||
| } | ||
|
|
||
| VPValue *NewData = nullptr; |
There was a problem hiding this comment.
In general, recipes should manage any VPValues via their operands; otherwise replaceAllUsesWith won't work as expected.
| VPValue *getVPNewData() { return NewData; } | ||
| }; | ||
|
|
||
| class VPConditionalScalarAssignmentDataUpdateRecipe final |
There was a problem hiding this comment.
It's difficult to tell what the semantics of the recipe are without comments; it's not clear to me why this cannot be just a select?
|
|
||
| class VPConditionalScalarAssignmentExtractScalarRecipe final | ||
| : public VPSingleDefRecipe { | ||
| SmallVector<PHINode *> PhisToFix; |
There was a problem hiding this comment.
Does the recipe need to hold references to IR Phis? Can it instead be an operand of the phis that need updating?
| Value *AnyOf = State.get(getOperand(2), /*NeedsScalar=*/true); | ||
| Value *MaskSel = | ||
| State.Builder.CreateSelect(AnyOf, WidenedCond, MaskPhi, "csa.mask.sel"); | ||
| cast<PHINode>(MaskPhi)->addIncoming(MaskSel, State.CFG.PrevBB); |
There was a problem hiding this comment.
Why is this needed? A recipe's execution shouldn't modify outside the IR generated by the reicpe.
Can we instead update MaskPhi via the VPlan Def-use chains?
| return ST->hasVInstructions() && | ||
| ST->getProcFamily() == RISCVSubtarget::SiFive7; |
There was a problem hiding this comment.
It would be great to not limit this to a single processor family. IIUC there's nothing target-specific about the feature and the cost model should ideally accurately compute the cost, without needing a separate TTI hook?
FindLast approach only works when things are monotonic. CSA works regardless of whether things are monotonic. Reductions kick in before CSA, so if we can vectorize using FindLast, we will vectorize it (and prefer it) compared to CSA. |
This is "The motivating example" rather than one I brought up.
If
I think such patterns are essentially extensions of "FindLast" reduction and should be developed as such, rather than being considered distinct unrelated patterns.
Note that by replacing
-- one which would also seem to benefit from sinking |
|
@ayalz, we are carrying future patches which should improve the performance of the loop by using mask logic instead of reductions inside the loop. I planned on posting it as a follow up since it is really more performant in cases where the target core has mask units that can handle it well. It looks something like this: This is not the same as a FindLast inside the loop because there is no reducing on each loop iteration. Since this pattern is not an extension of "FindLast", I'm not sure it is a good idea to develop CSAs as in loop reductions. I think we're saying the same thing though, where we both want to see FindLast in the middle block. This is essentially what I'm doing with the ExtractRecipe. I wrote the ExtractRecipe since FindLast only supports monotonics. I still think that we need to keep the in-loop approach though, regardless of whether we use FindLast outside the loop or not?
@Mel-Chen can you chime in here? Can FindLast handle non-monotonic cases? I think the reason we took the approach proposed in this patch was because FindLast only works for monotonic cases. |
I think what @ayalz wants to discuss with you is the newly added I still believe that the semantics you are implementing are a form of reduction. Therefore, reusing the existing reduction framework is appropriate. I don't think this change would affect how you ultimately generate the vectorized IR. Ideally, the vectorizer should select the best vectorization approach in the following order:
|
|
@michaelmaitland - thanks for the intriguing discussion!
It may be helpful to distinguish between:
The canonical pattern for the motivating example, presented with the following input pattern: is arguably that of a FindLast reduction as the canonical pattern. Indeed, an alternative input pattern is, using a sentinel: Hopefully these two input patterns could be recognized and manipulated using common infrastructure including recipes, and benefit from similar output patterns. Now, regarding the latter - multiple options may indeed be considered, with the one having the best cost to be selected. The above example demonstrates one such option, where As mentioned above, it may indeed be "better to pass vmask as a live-out and sink looking for its last turned-on lane to after Now, whether it is better to sink a single scalar load of are also two options to chose from based on cost, unless one clearly outweighs the other.
FindLast conceptually computes that last iteration for which some condition holds, and iteration count is conceptually monotone and increasing (wraparound at the end issue?). Computing FindLast using a sentinel value, however, requires that such a value exists. In its absence, an external "found" indicator can be used or the condition can be checked for the resulting iteration, as @Mel-Chen pointed out in several TODOs of #67812. Note that decreasing IV's are derived from the canonical iteration count IV, and their derivative function could be sunk as well. In any case, if the current limitation of FindLast regarding monotone/sentinel is too restrictive, would be good to work towards lifting it? |
Curious about there need for both FindLast and FindLastIV. The desired output may be some function of the last IV found, and this function may be applied after the loop or folded into the loop along with the reduction if preferred (similar to two invariants per boolean AnyOf), but the canonical reduction pattern for both is presumably that of finding the last iteration? Trying to reason about where the above ideal order may be important: every header phi, which represents a cross-iteration dependence, must be handled by the vectorizer, according to one of the following vectorizable categories:
There may be additional recurrences to consider? |
The reason for distinguishing FindLastIV from FindLast is similar to the distinction between AnyOf and FindLast. Under specific conditions—such as AnyOf for two invariants or FindLastIV for a monotonic sequence—we can achieve better vectorization performance by using the lower-cost AnyOf or FindLastIV instead of always using FindLast. While it is possible to initially classify all non-min/max select reductions as FindLast and later refine them into AnyOf or FindLastIV, I don’t see any clear advantage in doing so. Instead, performing a precise classification of AnyOf, FindLastIV, and FindLast during the legality check allows the target to determine whether to apply in-loop or out-of-loop reduction based on the specific reduction kind after in-loop non-min/max select reductions are supported in the future.
I'm currently considering adding MinMaxRecurrence to better support min/max with index idioms. The current reduction analysis does not allow internal loop users outside the recurrence chain, nor does it allow the recurrence chain without external users. This does not align well with the conditions of min/max reduction in min/max with index, which is why I'm not satisfied with my previous implementation. Other than the MinMaxRecurrence I'm planning, I haven't thought of any additional recurrences to consider. |
This patch adds initial support for CSA vectorization LLVM. This new class
can be characterized by vectorization of assignment to a scalar in a loop,
such that the assignment is conditional from the perspective of its use.
An assignment is conditional in a loop if a value may or may not be assigned
in the loop body.
For example:
Using pseudo-LLVM code this can be vectorized as
On each iteration, we track whether any lane in the widened condition was active,
and if it was take the current mask and data as the new mask and data vector.
Then at the end of the loop, the scalar can be extracted only once.
This transformation works the same way for integer, pointer, and floating point
conditional assignment, since the transformation does not require inspection
of the data being assigned.
In the vectorization of a CSA, we will be introducing recipes into the vector
preheader, the vector body, and the middle block. Recipes that are introduced
into the preheader and middle block are executed only one time, and recipes
that are in the vector body will be possibly executed multiple times. The more
times that the vector body is executed, the less of an impact the preheader
and middle block cost have on the overall cost of a CSA.
A detailed explanation of the concept can be found here.
This patch is further tested in llvm/llvm-test-suite#155.
This patch contains only the non-EVL related code. The is based on the larger
patch of #106560, which contained both
EVL and non-EVL related parts.