I'm  an  Instructor in  the  JMP  Education  group.

My  co-authors  for  this  presentation  are Jordan  Hiller  and  Byron  Wingerd,

who  are  both  JMP  Systems  Engineers.

We're  here  today  to  warn  you

of  the  dangers  in  using JMP's  Levey- Jennings  control  charts.

We  wanted  to  present  at  Discovery because  we  believe   Levey-Jennings  charts

are  being  misused, resulting  in  missed  signals.

Today,  I'll  start  with a  quick  control  chart  review

followed  by  a  quick  history,

and  that  will  lead  into  how  we've  seen Levey-Jennings  charts  being  misused.

We'll  conclude  with  some  recommendations

for  use  of  these  charts and  some  final  thoughts.

Now,  every  quality  system has  a  bit  of  process  control  in  it.

These  are  the  DMAIC  steps  from  Six  Sigma,

but  you  might  be  using  QbD or  another  quality  system.

Control  charts  are  used  at  the  end of  an  improvement  process

when  you've  put  the  process  on  target and  minimize  the  variability.

The  process  is  now  stable,

which  means  the  probability  distribution is  not  changing  over  time.

Now  it's  time  to  monitor that  process  variability

to  verify  that the  distribution  stays  the  same

and  is  not  changing  in  the  future.

What  is  a  control  chart?

A  control  chart is  simply  a  run  chart  of  the  data

with  the  addition  of  control  limits.

The  center  line  is  based on  the  historical  mean  of  the  process,

and  the  width  of  the  control  limits

is  based  on  the  historical variation  of  the  process.

If  the  process  distribution  does  change,

observations  will  eventually  fall outside  of  the  control  limits,

signaling  to  the  process  owner to  take  action  to  assign  a  cause

to  the  out-of-control  point  and  restore the  process  to  a  state  of  control.

Many  types  of  control  charts  have  been developed  for  different  situations.

XM-R  charts  or  individual and  moving  range  charts

for  data  collected  one  point  at  a  time.

Xbar-R  or  Xbar-S  charts for  summarized  data

collected  multiple  points  at  a  time.

CUSUM  or  EWMA  control  charts to  detect  small  shifts,

and  even  m odel-driven multivariate control charts

to  control  multiple  correlated variables  on  one  chart.

To  find  those all  important  control  limits,

you  have  to  answer  this  question,

how  do  you  estimate  the  historical process  distribution?

First,  you  have  to  determine if  the  process  is  stable  enough

for  control  charting.

Or  does  it  need active  process  improvement?

This  is  called  Phase  I.

It's  an  active  phase  of  data  collection and  sampling  plan  adjustment,

limit  calculation, maybe  even  some  statistical  modeling.

This  phase  ends when  you  have  collected  enough  data

to  determine  that  the  process  is  stable.

Then  the  SPC  system  shifts  into  Phase  II.

The  sampling  plan and  the  control  limits  are  fixed

and  new  data are  judged  by  the  fixed  limits.

There's  many  considerations for  Phase  I  data  collection.

The  sampling  plan, including  rational  sampling

and  rational  subgrouping,  the  sample  size,

how  much  data  do  you  need to  collect  to  fix  those  limits,

the  type  of  data  that  you're  collecting,

the  type  of  control  chart that  you  want  to  make.

Rather  than  talking about  all  these  in  detail  here,

I  will  refer  you  to  our excellent,  free,  online,

self-paced,  e-learning  course called  Statistical  Process  Control.

It's  available  in  the  JMP  Community.

You'll  often  hear  that  control  chart limits  are  three- sigma  limits.

This  means  that  we  need  to  estimate

the  standard  deviation  of  the  process from  our  data,  multiply  by  three,

and  then  add  and  subtract  from  the estimate  of  the  mean  of  the  process.

When   Walter Shewhart  developed the  first  control  charts  in  the  1920s,

he  realized  that, like  other  statistical  procedures,

he  needed  to  test for  signals  using  the  noise.

He  needed  to  compare the  between  subgroup  variation

using  the  within  subgroup  variation.

This  led  him  directly  to  X bar- R  charts,

where  the  average of  the  within  subgroup  range

is  used  as  an  estimate  of  sigma.

Specifically,  it  led  him  away  from the  naive  thinking

that  we  should  use  X bar  plus  or  minus three  times  the  sample  standard  deviation.

Don't  estimate  sigma using  s for  control  charts.

Let  me  say  that  again.

Don't  estimate  sigma  using the  sample  standard  deviation.

Oh,  and  here  it  is  in  red.

Don't  use  the  sample standard  deviation  to  estimate  sigma.

Why  not?

The  purpose  of  control  charts is  to  detect  a  signal  from  your  process,

detect  a  change in  the  process  distribution.

Using  the  sample  standard  deviation

aggregates  all  the  data, including  a  potential  signal,

and  this  will  inflate  the  estimate of  sigma  and  you  will  miss  signals.

That  takes  us  back  to  the  beginning.

If  the  purpose  is  to  detect  signals,

why  would  you  make  a  chart that  makes  you  miss  signals?

Just  don't  do  it.

But  the  default  method

for  estimating  sigma in  JMP's   Levey-Jennings  control  chart

is  to  use  the  sample  standard  deviation.

The   Levey-Jennings  chart  was developed  for  a  very  specific  situation

in  clinical  chemistry,

one  where  an  analytical  method is  validated  using  a  gage  study

with  factors  that  are  relevant to  that  process.

The  data  are  collected  according to  a  specified  experimental  design.

Here  are  some  possible  random and  fixed  factors  in  the  experiment.

The  variance  components for  the  random  factors

are  calculated  using  a  statistical  model.

Then  the  total  variability  is calculated  from  the  variance  components

and  used  as  the  estimate  of  sigma for  the   Levey-Jennings  control  chart.

This  chart  has  a  specific purpose  in  clinical  chemistry.

It  can  be  used  whenever  you  have an  external  estimate  of  sigma.

How  did  we  get  here  where  these  charts are  being  used  for  situations

in  industries  other  than the  one  I  just  described?

The  history  of  SPC  started about  100  years  ago

when   Walter Shewhart developed  the  first  control  chart.

He  figured  out  to  use  the  within or  the  short-term  variation

using  subgroups  with  multiple  items.

But  he  couldn't  figure  out  how  to  estimate

short-term  variation for  subgroups  of  size  1.

There  is  no  within.

In  1950,  Levey  and  Jennings published  their  paper,

and  they  did  not  use   s to  estimate  sigma.

Their  paper  introduced  control  charts to  clinical  chemistry

and  used Shewhart's  Xbar- R chart with  two  replicates,  so  subgroup  size  2.

Now,  just  after  that,  in  the  early  1950s,

Tippett  discovered  that  you  can  use

the  average  moving  range of  individual  values

as  a  short-term  estimate  of  sigma.

It's  not  perfectly  a  within  estimator, but  it's  short-term  enough

that  you're  not  aggregating over  all  the  data.

Very  quickly, ASTM  and  Western  Electric  published  books

and  standards  using  Tippett's  X- MR  chart.

But  clinical  chemistry didn't  hear  about  these  charts.

In  1952,  Henry  and  Segalove published  a  paper  using  s

to  estimate  sigma using  subgroups  of  size  1.

Don  Wheeler  indicates  that

XM-R  charts  were  not  used very  often  until  about  1980.

He  introduced  them  to  Deming,

and  Wheeler  and  Deming  taught  thousands of  clients  and  popularized  the  XM-R  chart.

It's  really  interesting  to  me  that they  weren't  popular  back  then

because  in  my  experience, which  started  in  the  early  '90s,

charts  for  subgroups  of  size  1  are much  more  popular  than  Xbar  charts  are.

All  right,  just  as  the  XM-R  charts were  gaining  popularity  in  manufacturing,

James  Westgard published  his  Westgard  rules,

and  wrote  his  popular  book, Basic  QC  Practices  for  clinical  chemistry.

He  calls  the  control  charts, Levey-Jennings.

They  were  added  to  JMP in  version  5  in  2003  by  customer  request.

Now,  ever  since  Jordan  and  Byron  and  I have  been  working  at  JMP,

we've  seen  examples  from  all  industries

of  the  misuse  of  the  default Levey-Jennings  chart.

But  now  you  know  better, and  you  will  never  use

the  sample  standard  deviation

to  estimate  sigma for  control  charts  ever  again,  right?

All right, I'm  going  to  turn  it  over  to  Jordan,

and  he'll  show  you  in  JMP the  problems  with  misusing  these  charts.

Thanks,  Di.

We  created  a  simulation  in  JMP  to  show  how

the   Levey-Jennings  chart,  when  misused, will  definitely  lead  to  missed  signals.

You  can  download  this.

It's  a  JMP  add-in that's  saved  in  the  JMP C ommunity

with  the  other  presentation  materials.

Here  we  show  two  control  charts on  the  same  data  set.

Here's  the  individual  chart  on  top,

and  the   Levey-Jennings chart  on  the  bottom.

Just  a  brief  note,  the  individual's  chart

is  usually  displayed with  the  moving  range  chart  underneath  it.

We're  omitting  that for  visual  simplicity  here.

I'll  also  mention that  the   Levey-Jennings  chart

is  drawn  in  the  way  we're  warning  against.

That  is,  these  are  not  historical  limits,

these  are  limits  calculated from  the  data  in  the  chart  itself.

The  30  data  points that  we  are  graphing  here

are  drawn  randomly from  a  standard  normal  distribution.

That  is,  the  mean  is  zero, standard  deviation  is  one.

Whenever  I  click this  new  data  button  over  here,

we'll  get  a  new  random  sample.

Because  we  know the  true  population  parameters,

we  can  tell  how accurately  these  charts  perform.

The  true  population  sigma  is  one, and  we're  going  to  hope

that  the  sigma  estimates from  the  two  charts  are  close  to  that.

The  data  is  stable, it's  from  a  normal  distribution.

Whenever  we  see  a  point outside  the  control  limits,

it's  a  false  positive  here.

In  a  simulation  of  5,000  data  sets,

the  individual's  chart has  a  false  positive  about   7%  of  the  time,

while  the   Levey-Jennings has  one  about  3.5 %  of  the  times.

This  small  difference,

well,  it's  double,  but  this  difference in  the  false  positive  rates,

we  think  is  one  of  the  reasons that  folks  have  tended  to  prefer

the   Levey-Jennings to  the  individual's  chart  sometimes.

But  here's  the  problem.

What  happens  when  we  have some  signals  in  this  data?

We're  going  to  talk  with  a  couple of  different  kinds  of  signals.

Let's  start  with  a  shift.

A  shift  is  an  abrupt  change in  the  mean  of  the  process.

Let's  start  by  looking  at a  shift  of  three  standard  deviations,

and  I'm  going  to  hit the  new  data  a  couple  of  times.

Remember,  the  purpose  of  a  control  chart

is  to  distinguish  the  signal from  the  noise  and  to  detect  signals.

As  the  signal  gets  larger, we  should  be  more  likely  to  detect  it,

and  the  sigma  estimates  should  remain close  to  the  true  value  of  one.

The  individual's  chart  does exactly  what  we  expect.

It  usually  detects  the  shift,

and  the  sigma  estimate  for the  individual  chart  stays  close  to  one.

The   Levey-Jennings,  not  so  much.

It  conflates  the  signal  with  the  noise.

The  sigma  estimate  is  inflated, and  signals  generally  can't  be  detected.

How  big  does  this  shift  have  to  be

before  the   Levey-Jennings  chart is  going  to  detect  it?

Well,  I'm  just  kidding.

It's  a  trick  question.

The   Levey-Jennings  chart  is  never going  to  detect  a  shift  in  this  scenario.

I  can  jack  it  up  to 50  standard  deviations,

and  the  larger  the  shift  that  we  induce,

the  more  that  sigma  estimate  is  inflated and  will  never  detect  a  shift.

Let's  see,  we  did  some  simulation  work

to  show  the  differences between  the  performance

of  the  individual's  chart  that's  in  red and  the   Levey-Jennings  chart  in  blue.

We're  looking  at  a  shift  that  occurs

after  the  20th  data  point in  the  series  of  30  data  points.

That's  what  we're  looking  at in  the  demo  I  just  showed.

In  that  situation, a  shift  after  the  20th  data  point,

you  can  see  the  performance of  both  of  these  charts

as  the  shift  gets  larger.

Individual's  chart  does  exactly what  we  hope  it  would  do.

As  the  shift  becomes  larger,

it's  easier  to  detect probability  of  having  an  alarm  is  greater.

The   Levey-Jennings performs  terribly  as  you  saw,

and  paradoxically, as  the  shift  gets  larger,

the  probability of  detecting  it  approaches  zero.

Let's  take  a  look at  that  here  in  the  demo.

We  can  change  the  shift  location,  right?

The  shift  was  after  the  20th  data  point.

But  you'll  see  that  as  the  shift approaches  the  end  of  the  series,

and  we'll  put  it  at  29, the  last  data  point  here.

When  the  shift  occurs at  the  last  data  point,

both  of  the  charts  perform  similarly.

In  other  words,  if  you  are  running your  Levey-Jennings  chart

after  every  additional  data  point, you'll  probably  detect  most  signals.

And  this  is  to  be  expected.

If  the  shift  occurs  later, there's  less  opportunity  for  that  signal

to  contaminate  the  noise  estimate,

and  the  sigma  estimate is  going  to  be  accurate.

Here's  what  that  looks  like in  the  simulations.

As  that  shift  gets later  and  later  in  the  series,

the  performance of  the   Levey-Jennings  chart  in  blue,

approaches  the  performance of  the  individual's  chart.

That's  the  story  with  shifts.

It's  even  more  alarming  with  drifts.

I'm  going  to  reduce the  shift  size  to  zero.

As  we  introduce  drift  into  the  data,

drift  is  a  gradual  linear  trend

as  the  mean  of  the  process moves  up  in  a  consistent  way.

That's  drift.

No  matter  how  large  the  drift that  I  induce  here,

it  will  never  be  detected by  the   Levey-Jennings  chart.

Just  can't  be  detected.

Okay.

N ext  let's  consider  a  batched  process.

Here,  we're  going  to  simulate

batches  with  size  six, five  batches  here  in  the  data.

Some  folks  use  the   Levey-Jennings  chart

as  a  way  to  avoid  having  an  alarm that's  due  to  expected  batch  variation.

Well,  it's  easy  to  see the  problem  with  this  approach.

When  we  overlay  a  shift  or  a  drift on  top  of  that  batch  effect,

the   Levey-Jennings  chart is  still  never  going  to  detect  it.

I'll  add  a  shift  too.

Yeah,  the   Levey-Jennings  chart is  insensitive,  too  insensitive,

and  the  individual's  chart is  too  sensitive.

Neither  of  the  charts is  really  great  for  this  situation.

Di  is  going  to  talk  about  this in  a  few  moments.

Before  I  turn  it  back  to  Di,

let  me  show  you  what

a   Levey-Jennings  chart done  right  looks  like.

This  is  a  demo  that  was  generated by  my  colleague,  Byron,

using  data  that  come from  the  Westga rd  website,

and  let's  just  talk  about maybe  this  top  chart  here.

We're  showing  a  chart based  on  a  data  series.

We  have  28  data  points.

The  mean  is  198.75  in  this  sample.

The  standard  deviation  is  5.9 in  this  sample.

The  point  is,  we  are  in  Phase  II,

and  as  such, we  have  set  the  control  limits

using  a  historical  estimate  of  sigma.

This  protects  us  against  all  of  the problems  that  we  were  just  discussing.

It's  a  stable  estimate  of  sigma,

and  it  will  not  be  contaminated by  any  potential  signals  in  the  data  set.

Okay,  that  is  all  I  have.

Di,  back  to  you.

All  right.

Thank  you,  Jordan,  for  that  alarming  demo.

When  Jordan  and  I  and  Byron

thought  that  it  was  time that  someone  gave  this  talk,

we  asked  our  JMP  friends if  they  were  seeing  what  we  were  seeing

from  talking  with  customers,

and  that  is, misapplication  of   Levey-Jennings  charts.

What  we  found  was  that  these  charts

were  commonly  used for  specific  situations.

The  first  one  is  what  Jordan  was talking  about,  that  batch  processing.

It's  called  short- run  SPC, where  you  might  have  batches

that  change  frequently, and  each  batch  has  a  new  mean,

maybe  even  a  new  standard  deviation.

Using  the  sample  standard  deviation

to  estimate  sigma will  contain  batch  shifts

as  well  as  within- batch  noise.

Like  you  saw,  you  won't  be  able  to  detect shifts  within  a  batch  very  easily.

Instead,  you  can  use  a  chart  that  plots

the  difference or  the  standardized  difference

from  the  batch  target  or  the  batch  mean.

The  second  situation is  autocorrelated  processes,

and  this  is  where  observations taken  close  together

are  more  similar  than observations  taken  further  apart  in  time.

This  behavior  is  often  seen  when measurements  are  taken  very  frequently

in  a  continuous  process.

Using  the  sample  standard  deviation,

again,  includes  both  noise and  known  process  drift

due  to  the  auto correlation in  the  estimate  of  sigma.

There  are  a  few  ways to  deal  with  autocorrelated  data,

including  reducing the  sampling  frequency  if  you  can,

or  using  a  different type  of  chart  if  you  can't.

Charts  like  CUSUM  and  EWMA can  be  adapted  for  autocorrelation,

or  you  could  try to  model  the  autocorrelation

and  then  use  control  charts on  the  residuals  from  the  model.

The  residuals  also  contain  information

about  process  shifts, and  they  should  be  uncorrelated.

F inally,  we've  heard  a  lot  of  people  say that  using  the  sample  standard  deviation

is  useful  because  it  gives  wide  limits that  are  able  to  detect  huge  shifts,

but  not  too  wide  to  detect small  and  moderate  shifts.

In  that  case,  we  recommend that  you  just  make  up  some  limits

and  don't  advertise that  you're  using  control  charts

because  control  charts  don't  ever  use

the  sample  standard deviation  to  estimate  sigma.

Remember  that  West gard's  wonderful  book

is  called   Basic  QC  Practices , not  Basic  SPC  Practices.

Feynman's  wonderful  quote is  applicable  here.

We've  also  heard  some  arguments about   Levey-Jennings  charts.

They  are  more  forgiving than  Shewhart's  charts.

Of  course,  they  are.

I  like  this  one.

The  range  charts were  optimized  for  hand  calculation,

and  we've  got  computers.

Why  aren't  we  calculating standard  deviation?

As  we've  seen, it's  not  range  versus  standard  deviation,

it's  which  standard  deviation?

You  should  always  choose within  variability

when  you  calculate   three-sigma  limits.

Aggregate  over  noise, not  signal  or  potential  signal.

Why  use  an  estimate  of  sigma when  we  can  just  calculate  sigma?

Oh,  this  one  hurts  me. This  comes  from  a  terminology  issue.

We  may  say  three- sigma  limits,

but  we  don't  know  sigma and  we  have  to  estimate  sigma  from  data.

This  is  the  way.

I  inherited  this  system, and  my  boss  says  I  have  to  do  it  this  way.

Well,  can  you  find  a  new  boss?

Really,  this  is  my  Oprah  moment.

Maya  Angelou  said,

"In  the  past, you  did  the  best  with  what  you  had,

and  now  that  you  know  better, you  will  do  better."

I  hope  that  you  can  educate others  to  do  better  as  well.

Thank  you  for  listening.

I've  got  some  references  for  you here  at  the  end  of  the  presentation.

We'd  like  to  leave  you  with  two  thoughts.

First,  don't  use  the   Levey-Jennings  charts

as  they  have  been  defined in  the  modern  world.

The  purpose  of  a  control  chart is  to  detect  process  changes,

and  those  changes  are  found by  comparing  signal  to  noise.

Use  of  the  sample  standard  deviation

to  estimate  sigma  inflates  that  noise and  it  will  obscure  any  signals.

The  second  thought  is  that  when  you  are in  Phase  I,  use  XM-R  charts,

individual  moving  range  charts instead  of   Levey-Jennings.

When  you  move  to  Phase  II

for  ongoing  process  control, fix  those  limits.

A  sign  of  a  stable  process

is  that  the  sigma  estimate from  using  the  average  moving  range

is  similar  to  the  sigma  estimate from   Levey-Jennings.