Hi, I'm Dave Trindade,
founder and owner of Stat-Tech, a consulting firm specializing
in the use of JMP software for solving industrial problems.
Today I'm going to to talk about a consulting project
that I worked on over the last year
with a robotics company.
We're going to be talking about Drones F lying in Warehouses:
An Application of Attribute Gauge Analysis.
Attribute gauge analysis is typically applied to compare agreement,
or lack thereof between two or more rating approaches to a problem.
For example, two inspectors may have differences of opinion
as to whether a part is conforming, call it pass
or nonconforming call it fail.
Based on consideration of specific quality indicators for individual parts,
how do we quantitatively measure the degree of agreement?
Let's actually start off with an example.
Let's say we have two inspectors,
inspector 1 and Inspector 2,
and they are presented with a list of 100 parts,
the critical characteristics on these 100 parts
and asked to determine whether each part should be classified as a pass or a fail.
I've summarized the results on the table to the right, partial table.
You see there are 100 rows in the table.
All variables are nominal, so the first column
is the part 1-100.
Then the second column is the rating by Inspector 1,
whether it's a pass or fail, and then the second inspector
where also second column has a pass or fail rating.
Now, if we were not familiar with JMP's gauge attribute analysis program,
first step that we could take could be
to look at the two classification distributions
and use dynamic linking to compare.
What I will do is show you the slides
and then I will go and demonstrate the results on the slides
after I've gone through the slides, through a certain amount of material.
For example, if we click, say, let's generate distributions
of the two columns, the fail, Inspector 1 and Inspector 2,
then we can click, say, for example, on the fail column for Inspector 1,
you see mostly matches for Inspector 2,
but there are a few disagreements over here.
There are some passes where Inspector 1
classified it as a fail, but Inspector 2 classified it as a pass.
Now, when you do click on that JMP highlights,
the actual rows that correspond to this.
You can see over here, for example, row four, Inspector 1 called it a fail
and Inspector 2 called it a pass.
Generally though, they're mostly in agreement,
fail, failed, fail, fail and so forth.
We could also do that by clicking on Inspector 2 fail,
and then seeing how it compares to Inspector 1.
We see that there are actually five instances of disagreement
between the two inspectors.
When the Inspector 1 classifies it as a fail,
there's five that I nspector 2 classified it as a pass.
Now we can also visualize the inspector comparison data.
To do that, we can use graph builder with tabulate to view agree and disagree
counts between the two inspectors. Here's one way of visualizing it.
We can put Inspector 1 using graph builder on the horizontal axis
and Inspector 2 on the vertical axis.
Then we see now with color coding, whether it's agreement or disagree,
agree is green and then the rows
that are disagree color code is red markers.
Now we can see the actual distribution.
Then we can use tabulate to actually total the numbers that are involved.
We can see over here for Inspector 1 and Inspector 2.
Inspector 1 and Inspector 2 agreed on the fail categorization
for 42 of the parts and they agreed on 44 of the pass parts.
They disagreed on nine instances over here
where Inspector 2 called it a fail and Inspector 1 called it a pass.
And Inspector 2 called a pass, where Inspector 1 called it a fail.
So those total 14.
The inspectors agreed on a classification for 86% of the parts
and they disagreed on 14%.
From there now we can go and do attribute gauge analysis
and see what JMP can do for this analysis.
To go to attribute gauge analysis, we're going to go to quality
and analyze quality and process variability attribute gauge chart.
Then we cast the rows. And here I've shown the inspectors
are listed under the Y response, both of them.
Then the column for the part is listed as the grouping.
These are required entries.
We notice the chart type is attribute, we click okay.
And now JMP provides us with this attribute gauge analysis report.
The first chart that's shown over here is the percent agreement for each part.
So we have 100 parts on the horizontal rows axis over here,
and when there's 100%, it means the two inspectors agreed.
When there's 0%, it means to disagree.
The left chart shows the overall percent agreement 86% by inspector.
Since the comparison is between only two inspectors,
both are going to have the same 86% agreement value.
The agreement report now includes
a numerical summary of the overall 86% agreement.
You can see 86 matches out of 100 inspected
and the individual ones are going to be the same since it's only one issue
of whether it was a pass or fail for a given part.
And 95% confidence limits are provided for the two results,
both for the inspector and for the agreement.
Now the agreement comparisons report includes a new statistic
that perhaps many people are not familiar with,
called the Kappa Statistic.
It's devised by Cohen, that's given in the reference.
The Cohen Kappa Statistic index, which in this case is 0.7203.
Is designed to correct for agreement by chance alone.
This was very interesting to me when I first read about this.
Like, "What do we mean by agreement by chance alone?"
Let's go into a little bit of an explanation of agreement by chance
and how can we estimate it.
Let's consider two raters, R1 and R 2.
We'll assume totally random choices
for each rater for each sample, example, for each part.
We further assume that the probability a rater selects either choice pass
or fail over the other is 50%. So it's 50/50.
Hundred samples or trials are differently categorized by pass/ fail for each rater,
similar to flipping a coin for each choice.
We just visualize two inspectors and they're each flipping a coin,
and they're trying to match how many get head, heads
or tail, tail, or head, tail or tail, head.
What's the expected fraction of agreements by chance?
Well, it's a simple problem in probability.
Similar to tossing two coins, there are only four possible
and equally likely chance outcomes between the two inspectors for each part.
Rater 1 could call it a fail, and Rated 2 could call it a fail.
They would agree. Rater 1 can call it a pass,
and rater 2 would call it a pass and there'd be agreement there.
The disagreement would be when they don't agree on whether it's a pass, fail.
Now, these are four equally likely chances.
Well, two of them are to agree and two of them are disagree.
Therefore, the probability of agreement by chance alone is two out of 4 it's 50%.
It's a simple probability problem.
Now, how do we calculate the Kappa S tatistic on the other hand?
As I say, it's meant to correct for this expected probability
of agreement by chance alone.
The simple formula that you can put into JMP for the Kappa Statistic
is the percent agreement, in this case, would be 86% minus the expected by chance
from the data, which we know is going to be around 50%.
How do we actually use the data to estimate the expected chance
from the data itself, the expected agreement by chance from the data?
Well, the estimation of Cohen Kappa Statistic
is shown below, and this is basically how it's done.
This is the tabulated value. This is what we saw earlier.
Agreement on fail/ fail for Inspector 1 and 42 instances.
Agreement between Inspector 1 and Inspector 2
on the past criteria for 44 instances.
You had those up and that's 86%.
So I show you over here in the Excel format that we added, 42 plus 44
we got divided by 100.
Then disagree is one minus [inaudible 00:08:58]
or just the five plus nine divided by 100.
Now, to calculate the agreement by chance, the Cohen Kappa Statistic is the sum
of the products of the marginal fractions for each pass/ fail type.
Well, here are the marginal fractions.
F or fail, the marginal fractions are 51 divided by 100 and 47 divided by 100.
So we form a product of those two. Fifty-one divided by 100
times 47 divided by 100.
Plus, now we enter the fail. The other criteria is the fail.
This is 49... I should have said that was the first disagreement.
Fifty-one and 47 for the fail criteria. For the pass criteria now,
we go 49 out of 100, and 53 out of 100,
and we multiply those two together and add them.
That gives us a number when we calculate it out to 49. 94,
which is very close obviously, to 0.5.
Then the Kappa Statistic is the percentage agreement
minus the expected by chance, divided by one minus the expected by chance.
That comes out to be 0.7023 in this case.
The Kappa. Here are some guidelines
for interpreting Kappa.
Go back up again.
I want to show you that the Kappa was 72.03.
The guidelines for interpreting Kappa are if Kappa is greater
than 0 .75, it's excellent.
If it's between 0.4 and 0. 75, it's good.
And if it's less than 0 .40 it's called marginal or poor.
There's some finite divide lines on this.
An excellent, totally 100% agreement,
would make a Kappa of one, and we could actually get a negative Kappa
which would be agreement that's less than by chance alone.
The books for these are given in the reference.
All right, let me just stop here and go into JMP to what we've done so far.
We can see that.
The data file that we've got over here is the inspectors
that I talked about, and I said we could take a look at the distribution.
Obviously, if you're familiar with JMP, you just go analyze distribution
and put in the inspectors over here.
Then we can put this next to the rows,
and if we click fail, we see the comparison between the fails
for Inspector 1 versus the fail for Inspector 2 and some disagreements.
Then you can see which areas the rows that disagree it's example four.
Similarly, if we calculate over there, we can compare Inspector 2 to Inspector 1.
Get a different set of numbers over here.
The other option that I mentioned for visualization is to do a graph builder
and in graph builder, we can put Inspector 1 on the horizontal axis
and Inspector 2 on the vertical axis.
Now we have a comparison, and now we can actually see the numbers
of times, for example, how many times did Inspector 1
rate something as a pass where Inspector 2 rated as a fail.
If we go back to the data table,
we see that there are nine instances of that, and they're highlighted in a row.
Th is is a very quick way of seeing what the numbers are
for the different categories that we're working with.
Let's say, click done over here.
The other thing we can use as I mentioned, is the tabulate feature.
We can go to tabulate, and we can put Inspector 1 on the top
and Inspector 2 on the bottom over here this way.
Then we can add in another row for the marginal totals down here and so forth.
Now we have the summary that we can put next to the graph builder
and see what the actual tabulations are that we've got for that.
That's something that we would do
perhaps if we were not familiar with JMPs program.
But let's use JMP now to do the analysis.
We're going to come over here, we're going to go to Quality and Process.
We're going to go to variability attribute gauge chart.
We're going to put in the inspectors over here.
We're going to put in the part.
Over here you notice it's required, attribute.
Okay, click okay, and now we have our output.
This output shows again for each part the agreement.
Zero means I disagree. 100% means I disagree.
This shows the rating between the two inspectors, 86%.
This is the summary, 86%.
Here's our Kappa index over here, and we have the agreement within rates.
This is kind of redundant in the sense
because we're only looking at one binary comparison.
Then further on down here, we can do the agreement by categories.
We can actually calculate how much is the agreement by the fail,
individually, or by the pass individually.
Okay, so that's how we would do it for a simple comparison.
But what if we now consider that the actual diagnosis
of the part was known or confirmed?
Let's go back into our PowerPoint's presentation.
I introduce a standard.
Okay, this is a measure of what we call effectiveness.
We're going to assume the correct part, classification was either known
or subsequently confirmed.
So this is the true correct diagnosis that should have been done on that part.
How accurate are the inspectors choices?
In other words, how can we compare how accurate each inspector was
to determine matching up with the true standard?
We set up that column in JMP, and now we can go through the process
that we said earlier of looking at a distribution.
For example, if we add in the distribution,
this time we include a standard, now we can click on pass,
and we can see the agreements between Inspector 1 and Inspector 2
on pass classifications.
You can see both of them had some misclassifications, some wrong diagnosis.
We can click on fail and do the same thing for the other category.
Then JMP will highlight those in the data table.
To do the attribute gauge analysis in JMP using the standard,
all we have to do now is enter standard into the dialog box as we did before.
This is the additional column.
The big difference now, and this is not highlighted in the manual,
is that under attribute gauge, we can now get a chart
that applies specifically to the effectiveness.
What we're going to do is unclick these agreement points on the chart,
and click instead the effectiveness points under attribute gauge.
When we do that, we get another chart that measures the effectiveness.
And this effectiveness has three ratings for it.
This gauge attribute chart now shows the percent agreement,
0, 50 %, or 100% of the two inspectors to the standard for each part.
A 0% implies both inspectors misdiagnosed their problem.
Seven events of that occurred.
A 50% signifies one of the inspectors got their correct classification,
and obviously, 100% means they both got it right.
Then the left chart shows the overall percent agreement
to the standard for each inspector.
We noticed that there was some slight difference between the two inspectors.
We now generate the effectiveness report
that incorporates the pass/ fail comparisons
to the standard for each inspector.
You can see Inspector 1 got 42 of the fails correct.
He got 43 of the passes correct, but he got 10incorrect of the fails,
call them passes.
And he got five of the passes incorrect.
I find this notation a little bit confusing.
I put it down at the bottom.
When we say incorrect fail,
that means a fail was incorrectly classified as a pass.
When we say incorrect pass, it means a pass was incorrectly
classified as a fail.
You can get your mind going in crazy ways
just trying to always interpret what's in there.
What I've done is I created my own chart to simplify things.
The misclassifications shows over here that 17 actual fail parts were classified
as pass, and eleven pass parts classified as fail.
So that's in the JMP output.
But what I said over here, and I'd love to see JMP include
something similar to this as a clear explanation of what's going on.
Inspector 1, Inspector 2, the standard is pass,
and then the correct classification as pass is 43 and 42.
The misclassified as fails are five and six.
Then over here, when the standard is the fail,
the correct choices by Inspector 1 and Inspector 2 is 42 and 45.
And the misclassified, so when it was fail,
10 of them were classified as pass and seven over here.
Now, understand, a fail part classified as pass
is a part that's a defective part going up.
That's called a miss.
On the other hand, a fail part that is t he actual pass part,
we call that basically produces risk, that's a false alarm.
And JMP uses those terms, false alarm and miss,
later on I'll explain that.
I like this chart because it seems to make a clear explanation
of what's going on.
Using graph builder, again we can view the classifications
by each inspector as shown over here.
A gain, you can highlight specific issues there.
JMP also allows you to define a conformance.
In other words, we said non- conforming is a fail and conforming is a pass.
That way we can take a look at the rate of false alarms and misses
in the data itself as determined by the inspectors.
We can see that the probability of false alarms for Inspector 1 was 0.1
and Inspector 2 is 0.125. The probability of misses,
okay, this means that we let it defect the park, go out,
was higher for Inspector 1 and Inspector 2.
I'll show how these calculations are done.
To emphasize this, a false alarm
occurs when the part is incorrectly classified as a fail,
when it is a pass. That's called a false positive.
The false alarm, the number of parts
that have been incorrectly judged to be fails,
divided by the total number of parts that are judged to be passes.
Now, that's where this calculation is done over here.
If I go up here, here's the pass is misclassified as fail,
so if I take five out of 48, I end up with that number 0.1042.
Now the next thing is a miss.
That part is incorrectly classified as a pass.
When it actually is a fail. That's a false negative.
In this case, we're sending out a defective part.
The number of parts that have been incorrectly judged to be passes
divided by the total of parts that are judged to be fails
is 10 out of 42 plus 10 is 0.193.
A gain, going back to this table,
these are the parts that are fail, but 10 of them were misclassified as a pass.
So the parts that should have been classified as they fail is 52.
Ten divided by 52 gives you that number of 0.1923.
So I like that table is easier to interpret over here.
The final thing about the conformance report
is you can change your conformance category,
or you can switch conform to non-conform.
You can calculate also an escape rate. And that is the rate that the probability
that a non-conforming part is actually produced and not detected.
To do that, we have to provide some estimate or probability
of non-conformance to the JMP program.
I put in like 10%, let's say 10% of the time we produce a defective part.
Given that we've produced a defective part, what's the probability
that 's going to be a miss and then escape?
And that's the escape rate. That's the multiplication of the two.
our process will produce times the probability of a missed process
produces fail part times the probability of a miss.
Now let's go into JMP again
and we're going to use the inspection with the standard.
I'm quickly going to go through this.
We do analyze distribution, again, put into three over here,
and now we can click on a standard down here
and then we can highlight, compare the Inspectors 01 and Inspector 2.
Another way to visualize it is to use graph builder as we've done before.
Then we can put Inspector 1 over here.
Let me do it this way.
And Inspector 2 can be on this side now.
Then we can enter the standard over here on this side.
And now we have a way of clicking and seeing what the categories
were relative to the standard.
That's a very nice little graph, and if you wanted to say,
"Okay, how many Inspector 1
versus classified as a pass when it was a fail.
Now we can bring that to a stand, and the rows are highlighted too.
Let's go into JMP. Now we're going to analyze
quality and process, variability attribute gauge chart,
recall and I'm going to add in the standard over here.
We click the standard. Now here's the issue.
JMP gives us the attribute gauge chart, but this was for the agreement.
What we'd like to measure is against the standard.
We come up here on the attribute gauge chart
and what we're going to do is unclick anything that says agreement.
And click on anything that says effectiveness.
There might be a simpler way to do this eventually
in the [inaudible 00:23:31] programming in JMP.
Now we have the effect on this chart, again, as I said, 50% means
that one of the inspectors got it right, 0% means they both got it wrong.
And we had the agreement report showing 86 that we've seen before.
But what we want to get down to is the effectiveness rating,
the effectiveness report.
And now we see that Inspector 1 was 85% effective.
Inspector 2 was 87% effective.
Overall it was 86% effective.
Here's the summary of the miss classifications.
And these are the ones that are listed over here.
As they say this terminology you need to understand that incorrect fails
were correct passes and incorrect passes were correct fails.
Then the conformance report is down here, we showed you how to do the calculation
and then we can change the conforming category by doing that over here.
Or we can calculate the probability of escape, escape rate,
by putting in some number in that estimates,
how often we'd expect to see a defective part.
I'm putting in over here point one,
click, okay.
And then JMP gives us the probability of non-conformance and the escape rates
as shown over here now for each inspector.
I was going back to now my PowerPoint presentation.
Now that we have a feeling for these concepts of agreement,
effectiveness and the Kappa index, let us see how we can apply the approach
to a more complex problem engage analysis called inventory tracking.
As part of a consulting project with the robotics company,
and It's Vimaan Robotics,
and by the way, there's some wonderful videos
if you click over here that shows the drones flying
in the warehouse and so forth
doing the readings and some of the results from the analysis.
As part of a consulting project I was introduced to the problem
of drones flying in a warehouse using optical character recognition
to read inventory labels and boxes and shelves.
In measurement system analysis MSA,
the purpose is to determine if the variability in the measurement system
is low enough to accurately detect
differences in product- to- product variability.
A further objective is to verify that the measurement system
is accurate, precise and stable.
In this study, the product to be measured via OCR
on drones is the label on the container stored in racks on a warehouse.
The measurement system must read the labels accurately.
Furthermore, the measurement system will also validate the ability to detect,
for example, empty bins, damaged item, counts of items in a given location,
dimensions, and so forth. All being done by the drones.
In gauge R&R studies, one concern addresses pure error,
that is the repeatability of repeated measurements on the same label.
Repeatability is a measure of precision.
In addition, in gauge R&R studies,
a second concern is the bias associated with differences in the tools,
that is, differences among the drones reading the same labels.
This aspect is called reproducibility, that's a measure of accuracy.
The design that I proposed
was a cross study in which the same locations in the warehouses,
in the bins are measured multiple times, that's for repeatability
across different bias factors, the drones for reproducibility.
The proposal will define several standards for the drones to measure.
The comparisons will involve both within- drone repeatability,
drone- to- drone agreement consistency,
and drone- to- standard accuracy.
The plan was to measure 50 locations 1-50,
and three drones will be used to measure reproducibility
that's drone- to- drone comparisons, and there will be three passes
for each location by each drone to measure repeatability.
Now, multiple responses can be measured against each specific standard.
So we don't have to have just one item and a standard.
We can have different characteristics.
The reading can be binary, that is, classified as either correct or incorrect.
And also the reading can provide status
reporting for a location, like the number of units, any damage units, and so forth.
Examples of different responses
are how accurately can the drones read a standard label?
Are there any missing or inverted labels?
Are the inventory items in the correct location?
Is the quantity of boxes in a location correct?
Are any of the boxes damaged? This would be something
that a human person would be checking as part of an inventory control,
but now we're doing it all with drones.
Here's the proposal.
I have 50 rows over here,
150 rows actually, because each location is being read three times by each drone.
So I have drone A, drone B, and drone C.
Then these are the results of a comparison to the standard.
We're classifying five standards, A,B,C,D and E,
and they're randomly arranged in here as far as the location goes.
And it's one characteristic specify each of the 50 locations.
S ince we're doing three readings, it's 150 rows.
T hree drones reproducibility, three passes for each location
and by each drone that's repeatability
and the standards are specified for each location.
I'm going to make an important statement over here that the data that I'm using
for illustration is made- up data and not actual experimental
results from the company.
We can start off with distributions and dynamic linking.
We can now compare the classification of the drones by standard.
We generate the distributions and then we click on say, standard A,
and we can see how many drones got that standard A right,
or whether any drones had further misdiagnosis.
Same thing if we can click on standard E, we can see drone A had a higher propensity
for misclassifying standard E and same thing with drone C.
Now the chart below shows how well
the drones agreed with each other for each location.
Here are the 50 locations and we're looking at the drones comparing.
Now when you're comparing a drone to other drones, you've got a lot of comparisons.
You're comparing maybe drone one to i tself three times.
You're comparing zone one,
drone one to zone two, times for each one of the measurements.
So it could have like a 1, 2, 3 for zone,
drone one and a one, two, three for zone.
Two, and you're comparing all possible combinations now of those drones.
That's why the calculations get a little bit complex
when you get multiple drones in there.
But you're doing a comparison.
This shows the agreement among all the comparisons.
Now we noticed that between zones the five and 10,
that for these locations the accuracy dropped quite significantly
and that prompted further investigation as to why?
It could have been the lighting, it could have been the location,
it could have been something else that was interfering with the proper diagnosis.
You see most of the drones are reading accurately 100%.
This is an agreement between the drones,
so they were agreeing roughly 90, 91% at the time.
And these are the confidence intervals for each drone.
So this told us how well the drones were comparing to each other.
Now we got agreement comparison, the tables will show the agreement values
comparing pairs of drones and drones to the standard.
And the Kappa index is given against the standard and repeatability
within drones and reproducibility are all excellent based on the Kappa Statistics
and agreement across even the categories is also excellent.
So we're comparing here drone A to drone B
drone A to drone C, drone B to drone C, all doing excellent agreement.
We're comparing here the drones to the standards, all an excellent agreement.
And then this is the agreement, just basically a summary of it.
Then this is the agreement by the different categories.
Now again, we can look at the attribute chart for effectiveness.
Same way we click out all the agreement check boxes
and then click on the effectiveness boxes.
We see again over here, that seven and eight had the lowest
agreement to the standard.
Again, that could have been something associated with the lighting.
It could have been something associated with some other issue there.
Then the overall agreement to the standard by drone,
you can see they're about 95%.
The drone is pretty accurate and they were pretty reproducible
and the repeatability was excellent.
This is the effectiveness report. Now this is a little bit more elaborate
because now we're comparing it for each of the five characteristic standards
and these are the incorrect choices that were made for each one.
Out of 150 possible measurements,
drone A measured 142 correctly, drone B 145 and drone C 140.
So effectiveness is the important one.
How accurate were the drones?
We can see that the drones are all running up around an average about 95%.
This appears to be highly effective.
Then we have a detail analysis
by level, provided in Misclassification report.
So we can see individually how each one of these drones compared to the different,
how each one of the different characterizations
were measured correctly or incorrectly by the drones.
This is the ones that are misclassifications.
A gain, let me go into JMP.
Oh, one further example I meant to show up over here.
You using graph builder,
we can view the classifications and mis classifications by each strong.
This is a really neat way of showing it. I wish JMP would include this possibly
as part of the output, but you can see where the misclassifications occur.
For example, for drone A, when you misclassify drone C,
most of them were classified correctly, but there are a few that were not.
These show the misclassifications.
I like that kind of representation in graph builder.
Now let's go back into JMP and we're going to do
attribute gauge analysis multiple with the actual experiment that was run.
Okay, so we're going to analyze distributions so we can do this.
We can compare the drones to the standard.
Again, we can just click on a standard and see how it compares across the drones.
We can also do an analyzed graph builder.
And we can put the zone drone A
and then drone B, and then drone C over here.
And then we can put the standard in there
and it shows very clearly what's happening with that.
B ut we can go and also into JMP and use the JMP quality and process
variability attribute gauge.
So we add the three drones in here, we add the standard,
and we put in the location and we get our gauge attribute chart report
showing that drones as far as the agreement with each other, we're at 90%.
This one has the most difficult locations to characterize.
Here are the agreement reports that I've shown you.
Drone A, Drone B and Drone C agreement with the other drones and with itself too.
Drone A to drone B, these are the Kappa values.
This is the measurement to the standard, all very high.
And then these are the agreement across categories.
And then for the effectiveness,
to get that graph that we like to see for the effectiveness report,
we take out the agreement over here and click on now the effectiveness.
We now have the effectiveness plot
on the tap that shows us how the drones agreed with the standard.
We now go back into the PowerPoint presentation over here.
Okay, to summarize w hat we've done over here,
the use of attribute gauge analysis allowed the company to provide solid data
on the agreement and effectiveness of drones for inventory management.
T he results are very impressive. Subsequent results reported
on the company's website show inventory counts to be 35%, faster inventory costs
reduced by 40% and reduced missed- shipments
and damage claims reduced by 50% compared to the previous methods.
In addition, the system generates what we call actionable data
for more accurate, effective, safer, more cost effective,
and faster inventory control.
Some excellent references over here is Cohen's original paper,
and book by Fleiss is excellent,
has a lot of detail, and also the book by Le is well done.
I thank you very much for listening. Have a good day.
